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ABSTRACT 


The Chairman of the Joint Chiefs of Staff established the Joint Capabilities 
Integration and Development System processes for acquisition of joint capabilities which 
are achieved through network-centric applications, services, enterprise systems, Family 
of Systems (FoS) and System of Systems (SoS). In many cases, advanced technologies 
must be matured simultaneously by multiple systems to support the degree of 
interoperability and/or integration required. Current DoD guidance with respect to 
technology development and assessment is focused on a acquisition of a system which 
operates relatively independently within a collection of other independent systems. 

An approach to technology development and technology readiness assessment of 
advanced technologies which support network-centric systems is required for successful 
development and fielding of network centric warfighting capabilities. Fundamental 
activities of technology maturation and assessments are the definition of a relevant 
environment and the ability to identify the critical technologies that provide for 
interoperable or interdependent functions. This paper proposes definitions for System of 
Systems and Family of Systems, degrees/levels of interoperability, and SoS Technology 
Readiness Assessment requirements and guidelines. SoS acquisition strategies are 
proposed to support program synchronization and SoS engineering activities which are 
key to successful development of net-centric Service and Joint capabilities. 
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I. 


INTRODUCTION 


A. PURPOSE 

The Chairman of the Joint Chiefs of Staff established the Joint Capabilities 
Integration and Development System (JCIDS) (Chairman of the Joint Chief of Staff, 
2007) processes for acquisition of joint capabilities which are achieved through network¬ 
centric applications, services, enterprise systems, Family of Systems (FoS) and System of 
Systems (SoS). In many cases, advanced technologies must be matured simultaneously 
by multiple systems to support the degree of interoperability and/or integration required. 
Current DoD guidance with respect to technology development and assessment is 
focused on a acquisition of a system which operates relatively independently within a 
collection of other independent systems. 

An approach to technology development and technology readiness assessment of 
advanced technologies which support network-centric systems is required for successful 
development and fielding of network centric warfighting capabilities. Fundamental 
activities of technology maturation and assessments are the definition of a relevant 
environment and the ability to identify the critical technologies that provide for 
interoperable or interdependent functions. A review of DoD guidance, industry and 
academic research shows that there are inconsistent definitions of these network-centric 
or so called Information Technology (IT) ‘systems’ and an undefined taxonomy with 
respect to degrees of interoperability. 

This thesis will propose SoS and FoS definitions and an interoperability 
taxonomy to be used in the context of technology development and assessment of SoS. 
Given the SoS definitions and a interoperability taxonomy, relevant environment 
definitions and guidance for identification of critical technologies will be proposed for 
SoS that would enable the proper technology development and acquisition strategies as 
well as effective assessment of these technologies. Included will be fundamental 
requirements and guidelines for SoS Technology Readiness Assessments (TRA) above 
and beyond the current requirements and guidance for system TRAs. 
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B. BACKGROUND 


The DoD uses advanced technologies to provide for a warfighting edge. 
Technology maturation is critical to successful development of systems on schedule and 
within budget while meeting capability requirements. Immature technology drives 
program schedule, cost, and performance risks at an increasing rate as a system is 
defined, designed, developed and deployed. The Government Accountability Office 
(GAO) has reported multiple times on DoD programs that have routinely used advanced 
technologies that lack the required maturity and led to programs experiencing significant 
cost overruns and delays. 

In the 1990s, the DoD adopted the National Aeronautics and Space 
Administration’s (NASA) Technology Readiness Levels (TRLs) (Mankins, 1995) as an 
approach to measure technology maturity and established guidance for technology 
development and assessment consistent with the level of DoD investment at a program 
acquisition milestone. Initially, NASA’s TRLs were primarily defined for hardware. 
DoD developed and provided guidance for system Technology Readiness Assessments 
(TRAs) based on these hardware TRLs. Over a five year period, DoD expanded this 
guidance to include software, manufacturing, and biomedical TRLs. All DoD acquisition 
programs are required by DoD policy to have technologies matured to a TRL 6 
(system/subsystem model or prototype demonstrated in a relevant environment) prior to 
program initiation at Milestone (MS) B. A successful MS B authorizes a program to 
enter the System Design and Development (SDD) phase and commits DoD resources to 
development, production and fielding of a system or capability. This policy was often 
not enforced. In cases where the technologies were immature, approved technology 
maturation plans were often required to show how the technologies would be matured. 
DoD programs continued to experience delays and cost increases due to design and 
development of the system with immature technologies. 

In 2006, Congress passed legislation (United States House of Representatives, 
2006) that required the Milestone Decision Authority for Major Defense Acquisition 
Programs (MDAPs) and Major Automated Information Systems (MAIS) to certify 
(among other things) that all technologies had reached a TRL 6 with respect to maturity 
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prior to the MS-B. If certification was waived, Congress required the MDAP to provide 
a justification based on national security needs Also in 2006, The Nunn-McCurdy Act 
(United States House of Representatives, 2006) which provides for an exception 
reporting system on DoD MDAP unit costs starting at MS B was revised by the FY06 
National Defense Authorization Act (NDAA) to have programs report against the 
Acquisition Program Baseline (APB) Original Baseline Estimates vice rebaselined 
estimates for near breach (+15%), significant (+30%) and critical (+50%) cost overruns. 
These two pieces of legislation make it imperative that a program carefully and 
thoroughly assess and select mature technologies appropriate to the expected 
operationally relevant environment to mitigate delays and cost overruns associated with 
using immature technologies. A network-centric operational environment will be more 
stressing than that of a system-centric operational enviromnent. Assessment with the 
current DoD TRA independent system-centric guidance may fall short when used to 
conduct and certify technology maturity for SoS. It behooves all acquisition programs to 
manage technology risk appropriately given that at MS-B the APB metrics are put in 
place that establish maximum thresholds per Nunn-McCurdy for DoD acquisition. 

C. DISCUSSION 

Technology readiness assessments provide an indication of level of risk to the 
development of a system and are conducted in support of technology selection, system 
engineering and program management activities. TRLs are defined levels of maturation 
from basic science through technology prototyping, development and operational 
deployment of a system. Two fundamental activities for technology assessment are a 
definition of the relevant environment and the selection of Critical Technology Elements 
(CTEs). 

A technology element is ‘critical’ if the system being acquired depends on 
this technology element to meet operational requirements with acceptable 
development cost and schedule and with acceptable production and 
operational costs and if the technology element of its application is either 
new or novel (Deputy Under Secretary of Defense for Science and 
Technology (DUSD(S&T), 2005). 
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CTEs include software and hardware technologies, algorithms, methods, 
materials, procedures and techniques. CTEs drive functional and non-functional 
performance. Examples of non-functional CTEs would be those that are required for test 
and evaluation, manufacturing, and/or logistics support. 

A relevant environment is a set of stressing conditions representative of 
the full spectrum of relevant operational employments, which are applied 
to a CTE as part of a component (TRL 5) or system/sub system (TRL 6) 
model or prototype in order to identify whether any design changes or 
fixes are needed to support the required (threshold) functionality 
(Mandelbaum, 2007). 

The relevant environment for network-centric systems includes the 
interoperability or integration drivers necessary to a specific warfighting capability. The 
absence of agreed to definitions of network-centric systems such as enterprise systems, 
FoS or SoS confounds the ability for technologists to define a relevant environment in 
which to conducting a technology assessment and the identification of the appropriate 
CTEs for a capability development. 

The Carnegie Melon’s Software Engineering Institute (SEI) defines 
interoperability as: 

the ability of systems, units, or forces to provide services to and accept 
services from other systems, units, or forces and to use the services 
exchanged to enable them to operate effectively together (Kasunic and 
Anderson, 2004). 

Note that services are more than just connectivity - there is an implied quality, 
timeliness, and adherence to specified business processes. SEI’s technical note on 
measuring systems interoperability (Kasunic and Anderson, 2004) defines aspects of 
technical interoperability (or integration); technical interoperability places detailed 
demands at multiple levels, which range from physical interconnection to correct 
interpretation by applications of data provided by other applications. Dimensions of 
technical interoperability include sensors generating bits of information, communication 
channels transmitting the bits of information, computers processing the bits of 
information and weapons directed by messages composed of bits. 
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Integration is generally considered to go beyond mere interoperability to 
involve some degree of functional dependence. For example, a mission 
planning system might rely on an external intelligence database; an air 
defense missile system will normally rely on acquisition radar. While 
interoperable systems can function independently, an integrated system 
loses significant functionality if the flow of services is interrupted. An 
integrated family of systems must of necessity be interoperable, but 
interoperable systems need not be integrated (Kasunic and Anderson, 

2004). 

There are technical and systemic challenges with a SoS TRA being conducted as 
a system TRA. Technical challenges include a) Capability requirements and functional 
analysis should occur prior to specific system requirements, system functional analysis, 
and system technology development; however, many SoS are assembled from legacy 
systems and network-centric functionality may be constrained, b) Key Performance 
Parameters (KPPs) for a capability are not easily allocated to individual systems and their 
subsystems, c) appropriate SoS relevant environment modeling and simulation and test 
and evaluation environments will typically be built post system design and development, 
d) identification of critical technology elements given the interoperability or integration 
may not be obvious within a (re)composable context or environment and e) SoS are 
typically enabled with software which is easily changed incrementally over time. 

Systemic challenges within the DoD include: a) critical technology developed by 
the individual programs are in alignment with their respective schedules not the SoS 
program schedule, b) SoS technology selections and development prior to completion of 
capability engineering and then individual system(s) engineering drives up risk; SoS 
engineering needs to be at least through System Functional Review prior to a MS B 
decision, and c) it's challenging to test the critical technologies in an integrated manner if 
the individual systems have not had the opportunity to all develop their systems enough 
to have representative systems for SoS testing (e.g., relevant environment for a integrated 
heterogeneous distributed system) and d) the fielding of a SoS capability is typically 
time-phased over several years in capability spirals or increments with differing sets of 
systems and services. 
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D. RESEARCH QUESTIONS 


The following questions are appropriate when assessing SoS. 

1. What are the appropriate definitions of SoS in the context of conducting 
TRAs? 

2. What are the appropriate definitions for interoperability and its use in 
defining the operational relevant environment for conducting SoS TRAs? 

3. What is the approach for determining critical technology elements for 
SoS? 

4. What are the fundamental requirements and guidelines for conducting a 
SoS TRA and how are these different from a system TRA? 

5. What technology development and acquisition strategies should be 
employed for technology maturation for SoS given the challenges of 
synchronization of individual system acquisition schedules? 

6. When is the ‘right’ time to hold SoS acquisition milestones given the 
synchronization issues with the individual systems that make up the SoS? 

E. BENEFIT OF STUDY 

This study will benefit Science and Technology (S&T), Acquisition professionals 

and Senior Executives in the DoD in the conduct of TRAs in support of SoS acquisition. 

F. SCOPE 

The thesis will focus on SoS TRAs. SoS definitions, an interoperability 

taxonomy and relevant SoS environment definitions and guidance for identification of 
critical technologies will be proposed that will enable the proper technology development 
and acquisition strategies will be defined. This thesis will recommend additional 
requirements and guidelines for SoS TRAs above and beyond the current requirements 
and guidance for system TRAs. 

This thesis is scoped to address technology maturity only. A distinction is made 
in this thesis between technology maturity and a technology’s readiness to be transitioned 
(transitionability). Technology maturity is defined as the technology’s state or condition 
with respect to full/complete development as required to be emplaced in a system and 
provide for a specified functionality and perfonnance. Maturity can not be used as the 
only selection criteria for technology; technology needs to be assessed in the context of 
the total capability development over time and the acquisition strategy of said capability. 
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Technology maturity is by definition considered as one aspect of technology’s readiness 
to be transitioned. Technology transitionability is measured as a function of the 
technology’s maturity, availability (program has access to the technology), alignment of 
technology and program schedules, and sufficiency of funding to develop/modify/insert 
programmatically into the system. 

G. METHODOLOGY 

Qualitative methods are used in this thesis. Content analysis and participant 
observation are performed on DoD and DoD industry, non-DoD industry, and academic 
sources regarding system and SoS acquisition, interoperability and Integration, TRAs, 
and system and SoS engineering. Analysis of successful and failing SoS acquisitions is 
performed to determine how technology readiness assessments supported or failed to 
support their acquisition. These materials are synthesized into a concise articulation of 
requirements and guidance for SoS TRAs. 

H. ORGANIZATION OF STUDY 

The plan of this thesis is as follows: Chapter II provides an overview of literature 
on the topic of technology readiness assessments as well as related literature on SoS and 
interoperability, Chapter III synthesizes the literature review, Chapter IV provides 
preliminary analysis for SoS TRAs, and finally, Chapter V gives the summary, 
conclusions and recommendations for future actions and research regarding SoS TRAs. 
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II. LITERATURE REVIEW 


A. APPROACH 

This literature review encompasses system(s) definitions, interoperability 
definitions, selected ‘system’ examples and an overview of technology readiness 
assessments. Sources include books, articles, reports, and briefings from government, 
industry and academia sources. This literature review supports the content analysis 
research method for this SoS TRA topic. A SoS definition will be explored as a 
foundation for SoS technology readiness assessment. Interoperability will be explored to 
assist in identification and types of CTEs. Finally, technology readiness assessments will 
be analyzed within a view towards SoS TRAs. The following content analysis questions 
are answered at the beginning of each section: 

1. What data was analyzed? 

2. Why was this data identified to by analyzed? 

3. What is the domain from which it was drawn - DoD/government, non- 
DoD industry or academia? 

4. What is the context relative to which the data are analyzed? 

5. What are the boundaries of the analysis? 

6. What is the target of the inferences? 

B. SYSTEMS 

There are a variety of ‘systems’ from subsystems, systems, family of systems, 
system of systems, and enterprise systems. Network-centric systems require the 
connection of systems and may lead to some sense of unboundedness. The first step to 
defining a system is to delineate the boundary. The DoD requirements process and the 
nature of joint warfare for a specific mission area or task drives the scope of system 
connections. Without defining clearly the boundary of a system and what is to be 
developed, evaluated and deployed there may be less performance in a network-centric 
force than that of a system-centric one. 

Concise definitions of SoS type are useful in identifying critical functions and the 
technologies required to enable these functions. The scope of the operationally relevant 
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environment is constrained by the scope of the boundary at which to measure 
performance with respect to specified KPPs. The boundary of SoS is proposed to be 
encompassing a number of systems; therefore, architectural artifacts will be reviewed. 

A review of the DoD, industry, and academic literature finds multiple definitions 
for SoS or IT systems. Given a definition of the degree of interoperability, these SoS 
definitions may become clear. This literature review will review the tasks and/or 
missions the systems provide for and the degree of interoperability required to support 
defining SoS and the types of SoS if appropriate. 

DoD related literature is useful given the context of joint warfighting. Academic 
research is focused toward advanced concepts and technologies which may be applied in 
the future, whereas, commercial industry will provide the perspective of ubiquitous and 
diverse systems of all types (e.g., financial, medical) being developed and used globally. 

1. System of Systems Government/DoD Industry Literature Summary 

Prior to defining SoS, one needs to define system given at some level the SoS is a 
‘system’. In reviewing DoD literature there were numerous and varying definitions for 
‘system’. DoD-STD 480A defines system as follows: 

A composite of subsystems, assemblies (or sets), skills, and techniques 
capable of performing and/ or supporting an operational (or 
nonoperational) role. 

The JCIDS is governed by the Chairman of the Joint Chiefs of Staff Instruction, 
CJCSI 3170.OIF, latest dated 1 May 2007. This instruction defines: 

Joint concepts-centric capabilities identification process that will allow 
joint forces to meet the full range of military operations and challenges of 
the future. Meeting these challenges involves a transformation to a fully 
integrated, expeditionary, networked, decentralized, adaptable and lethal 
joint force able to achieve decision superiority...Potential solutions may 
include a family of systems (FoS) that take different approaches to filling 
the capability gap, each addressing operational considerations in a 
different way. Alternatively, the solution may require a system of systems 
(SoS) approach to fill a capability gap. The FoS and SoS materiel 
solutions may also require systems delivered by multiple sponsors and 
materiel developers...Capability Description Documents (CDDs) and 
Capability Production Documents (CPDs) developed in accordance with 
this instruction will be accepted to support capability development. 
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A CDD may not define the allocation of KPPs to individual systems. Capability 
is defined as “the ability to achieve a desired effect under specified standards and 
conditions through combinations of means and ways to perform a set of tasks.’ See 
Figure 1 for a perspective on capability-based system engineering. 

The CJCSI 3170 defines: 

Family of Systems - A set of systems that provide similar capabilities 
through different approaches to achieve similar or complementary effects. 

For instance, the warfighter may need the capability to track moving 
targets. The FoS that provides this capability could include unmanned or 
manned aerial vehicles with appropriate sensors, a space-based sensor 
platform or a special operations capability. Each can provide the ability to 
track moving targets, but with differing characteristics of persistence, 
accuracy, timeliness, etc. 

The CJCSI 3170 defines: 

Net centric - Relating to or representing the attributes of net-centricity. 
Net-centricity is a robust, globally interconnected network environment 
(including infrastructure, systems, processes and people) in which data is 
shared timely and seamlessly among users, applications and platforms. 

The CJCSI 3170 defines: 

System of Systems - A set or arrangement of interdependent systems that 
are related or connected to provide a given capability. The loss of any part 
of the system will significantly degrade the performance or capabilities of 
the whole. The development of a SoS solution will involve trade space 
between the systems as well as within an individual system performance. 
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Figure 1. Capability-Based System Engineering [From: Siel, 2006] 

Given the above definitions, one may come to the conclusion that it holds true 
only for a system of systems that if one system fails that the whole degrades. This may 
or may not be true given the robustness of the number of systems in the SoS. In fact, one 
may consider that for a minimized FoS that degradation may be as likely to occur where 
each ‘family’ member has a complementary mission to do that none of the family 
member have a capability to perform as well. 

From the DoD Guidebook (Under Secretary of Defense for Acquisition, 
Technology and Logistics (USD(AT&L), 2006) Chapter 4.2.6 on Systems Engineering: 

A family of systems does not create capability beyond the additive sum of 
the individual capabilities of its member systems. A family of systems is 
basically a grouping of systems having some common characteristic(s). 

For example, each system in a family of systems may belong to a domain 
or product lines (e.g., a family of missiles or aircraft). A family of 
systems lacks the synergy of a system of systems. The family of systems 
does not acquire qualitatively new properties as a result of the grouping. 

In fact, the member systems may not be connected into a whole. 
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From the DoD Acquisition Guidebook (DAG) (USD(AT&L), 2006) Chapter 4.2.6 
on Systems Engineering- this definition is the same at the definition in the CJCSI 3170: 

A system of systems is a set or arrangement of interdependent systems that 
are related or connected to provide a given capability. The loss of any part 
of the system will significantly degrade the performance or capabilities of 
the whole. The development of a system of systems solution will involve 
trade space between the systems as well as within an individual system’s 
performance. 

DoD has a draft Systems of Systems (SoS) Systems Engineering Guide: 
Considerations for Systems Engineering in a System of Systems Environment, version .9 
dated Dec 22, 2006 (USD(AT&L), 2006). It provides for extensions of traditional system 
engineering processes; however, it doesn’t distinguish between SoS and FoS. The guide 
defines a system as ‘an integrated composite of people, products, and processes that 
provide a capability to satisfy a stated need or objective’ and ‘a capability is the ability to 
achieve a desired effect under specified standards and conditions through combinations 
of ways and means to perform a set of tasks (citing CJCSM 3170.01B, May 11, 2005 - 
note: no change in the 01 May 07 version).’ It then defines SoS as: 

a set or arrangement of systems that results when independent and useful 
systems are integrated into a larger system that delivers unique capabilities 
(USD(AT&L), 2006). When integrated, the independent systems can 
become interdependent, which is a relationship of mutual dependence and 
benefit between the integrated systems. Both systems and SoS conform to 
the accepted definition of a system in that each consists of parts, 
relationships, and a whole that is greater than the sum of the parts; 
however, although an SoS is a system, not all systems are SoS. 

The guide states: 

For the SoS to function, its constituent systems must be integrated to 
achieve not only physical connectivity, but interoperability at all levels, 
including physical, logical, semantic, and syntactic interoperability. 
Interoperability allows the necessary connectivity across the SoS to be 
defined. 

The guide goes on to state: 

The boundary of any SoS can be relatively ambiguous because of the 
dynamic operational focus, multi-mission, and often ad hoc nature of the 
operational environment of the SoS. In this type of environment, there is a 
potential for ad hoc coupling across both organizational and systems 
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boundaries in support of the dependencies created. Therefore, in order to 
use systems successfully, in a SoS context, the protocols used to support 
the specification of interfaces should be ubiquitous because they are key 
convergence points for SoS and there may be no opportunity for changes 
to the interfaces without major impact to the entire SoS. The development 
and management of a SoS architecture through the evolution of an SoS is 
the mechanism used to document and share information among constituent 
systems to support integration. 

2. Non-DoD Industry Literature Summary 

The Institute of Electrical and Electronics Engineers, Inc. (IEEE)’s definition of 
system is: 

a set of functional elements organized to satisfy user needs (IEEE, 1994)” 
and/or “a collection of components organized to accomplish a specific 
function or set of functions (IEEE, 2002) 

Commercial industry literature including the International Council on System 
Engineering (INCOSE) has comparatively very little written about SoS and FoS from any 
organizations other than DoD and DoD Industry. 

In industry family of systems refers to a system that has several ‘variants’ that a 
similar to each other. They are not necessarily ever connected to work together. 

3. Academia Literature Summary 

Maier and Rechtin defines a system as: 

a collection of things or elements which, working together, produce a 
result not achievable by the things alone (Maier and Rechtin, 2002). 

A system of systems is described by Maier and Rechtin as systems which are 
operationally independent, managerially independent, evolutionary developed, with 
emergent behavior and are geographically distributed. The following definitions apply 
(Maier and Rechtin, 2002): 

Operational Independence of the Elements: If the SoS is disassembled into its 
component systems the component systems must be able to usefully operate 
independently. The SoS is composed of systems which are independent and useful in 
their own right. 
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Managerial Independence of the Elements: The component systems not only can 
operate independently, they do operate independently. The component systems are 
separately acquired and integrated but maintain a continuing operational existence 
independent of the SoS. 

Evolutionary Development: The SoS does not appear fully fonned. Its 
development and existence is evolutionary with functions and purposes added, removed, 
and modified with experience. 

Emergent Behavior: The system performs functions and carries out purposes that 
do not reside in any component system. These behaviors are emergent properties of the 
entire SoS and cannot be localized to any component system. The principal purposes of 
the SoS are fulfilled by these behaviors. 

Geographic Distribution: The geographic extent of the component systems is 
large. Large is a nebulous and relative concept as communication capabilities increase, 
but at a minimum it means that the components can readily exchange only infonnation 
and not substantial quantities of mass or energy. 

Maier and Rechtin goes on to describe three different types of SoS, Virtual, 
Voluntary, and Directed, and states not all SoS of similar complexity and extent should 
be regarded as equivalent. An additional dimension, that of managerial control, is stated 
as critical to identifying the types of SoS. The three basic SoS driven by managerial 
control as defined by Maier and Rechtin are as follows (Maier and Rechtin, 2002): 

Directed: Directed systems are those in which the integrated SoS is built and 
managed to fulfill specific purposes. It is centrally managed during long term operation to 
continue to fulfill those purposes, and any new ones the system owners may wish to 
address. The component systems maintain an ability to operate independently, but their 
normal operational mode is subordinated to the central managed purpose. For example, 
an integrated air defense network is usually centrally managed to defend a region against 
enemy systems, although its component systems may operate independently. 

Collaborative: Collaborative systems are distinct from directed systems in that 
the central management organization does not have coercive power to run the system. 
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The component systems must, more or less, voluntarily collaborate to fulfill the agreed 
upon central purposes. The Internet is a collaborative system. The IETF works out 
standards, but has no power to enforce them. Agreements among the central players on 
service provision and rejection provide what enforcement mechanism there is to maintain 
standards. The Internet began as a directed system, controlled by the Advanced Research 
Projects Agency, to share computer resources. Over time it has evolved from central 
control through unplanned collaborative mechanisms. 

Virtual: Virtual systems lack a central management authority. Indeed, they lack a 
centrally agreed upon purpose for the SoS. Large scale behavior emerges, and may be 
desirable, but the supersystem must rely upon relatively invisible mechanisms to maintain 
it. A virtual system may be deliberate or accidental. Familiar examples of what is called 
here a virtual system are the World Wide Web and national economies. Both ‘systems’ 
are distributed physically and managerially. The World Wide Web is even more 
distributed than the Internet in that no agency ever exerted real central control. Control 
has been exerted only through the publication of standards for resource naming, 
navigation, and document structure. Web sites choose to obey the standards or not at 
their own discretion. The system is controlled by the forces that make cooperation and 
compliance to the core standards. The standards do not evolve in a controlled way; rather 
they emerge from the market success of various innovators. National economies and the 
social ‘systems’ that surround us might be thought of as virtual systems. Politicians 
regularly try to architect these systems, sometimes through forceful means, but the long¬ 
term nature is determined by highly distributed, partially invisible mechanisms. 

Dr’s. Boardman and Sauser (Boardman and Sauser, 2006) describe differentiating 
characteristics of a SoS as: 

autonomy exercised by the constituent systems in order to fulfill the 
purpose of the SoS, constituent systems choose to belong to the SoS for 
greater good, SoS are typically connected dynamically to enhance the SoS 
performance, and characterized by a diversity of systems. 

Also of concern, SoS may seem unbounded. The levels of connectivity, platform 
diversity and degree of associated interoperability points to the risk of whether the SoS is 
unbounded. Bounded systems are characterized by centralized data and control, systems 
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and their linkages are known a priori and are specific to the connection and 
interoperability specified. Unbounded systems are characterized by protocols which 
provide a loose coupling and are omnipresent and allow for dynamic spontaneous 
connections (DiMario, 2006). 

Most SoS literature focuses on enabling interoperability via integration and 
therefore, focuses on architecture first. Remembering that interoperability is concerned 
with connectivity, capacity, correctness, accuracy, bandwidth, data latency, syntactic 
compatibility, consistency, completeness and undesirable semantic emergent behavior as 
cited above, we look at the architecture products developed during system engineering 
activities. 

Systems engineering methods provide a basis for exploring system 
interoperability. Figure 2 shows the Operational, System, and Technical architecture 
related views and their relationships that get created during the system engineering 
processes (Habayeb, 2005). 

Concept of operations within the context of an existing or to-be Enterprise 
architecture and mission requirements provide constraints and restraints on architecture, 
system development, and degrees of interoperability. The operational view provides for 
information exchanges, types of interoperability, and KPPs required to support a mission. 
The systems view defines system attributes, and provides the basis for comparing system 
performance against operational requirements. The technical view defines the standards 
and protocols to be implemented by the system for interoperability. 
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Figure 2. Linkages Among Architectural Views [From: Habayeb, 2005 ] 

Taking a deeper look into the Operational Views (OV) artifacts (see Figure 3): 

• The OV-1 provide a description of the operational concept 

• The OV-2 identifies the operational nodes, operational activities at each 
node, and the information exchanges needed between nodes. 

• The OV-3 identifies the information exchanges between nodes 

• The OV-5 identifies capabilities, relationships among activities and inputs 
and outputs. 

• The OV-6 describes the sequencing and timing of activities as well as 
business rules and processes. 

• The OV-7 documents the data requirements and business rules. 

System views (as seen in Figures 4 and 5) provide the detailed information 
regarding functionality required and the interfaces needed to enable this functionality. 
Taking a deeper look into the System Views (SV) one finds: 

• The SV-1 identifies the system nodes and interconnections between the 
nodes. 

• The SV-2 defines the communications architecture. 

• The SV-3 describes the interfaces. 

• The SV-4 documents the system functions and the data flow between 
them. 

• The SV-5 maps functions and operational activities 

• The SV-6 documents the data element exchanges 
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• The SV-7 documents the performance characteristics including the 
timelines. 

• The SV-11 documents the physical implementation e.g., messages 

The Technical Views (TV), TV-1 and TV-2 are used to represent current and 
future standards. 


Operational View 
Nine Views 


Framework 

Product 

Product Name 

General Description 

OV-1 

Graphic high Level 
ODerational ConceDt 

High-level graphical & textual description of operational concept 

OV-2 

Operational node 
Connectivity Descriptor 

Operational nodes, operational activities at each node, 
connectivity and information exchange need-lines between nodes 

OV-3 

Operational Information 
Exchange Matrix 

Information exchanged between nodes, and the relevant 
attributes of that exchange 

OV-4 

Organizational 
Relationship Chart 

Operational role, or other relationships among 
oraanizations 

OV-5 

Operational Activity 
Model 

Capabilities, operational activities, relationships 
among activities, inputs, and outputs. May show cost. 

OV-6a 

Operational Rules 
Model 

Describes sequence and timing of operational activities - 

identifies i: usmess rules that constrain operation 

OV-6b 

Operational State 
Transition Description 

Describes sequence and timing of operational activities 
identifies business process response to events 

OV-6c 

Operational Event 
Trace Description 

Describes sequence and timing of operational activities 
or sequence of events 

OV-7 

Logical Data Model 

Data requirements documentation and structural business 
process rules of the Operational View 


Figure 3. Operational Views OV-1 to OV-9 [From: Habayeb, 2005] 

It’s useful to now look (see Figure 6) at these architectural views and translate 
them into system engineering views to facilitate looking at inputs and outputs and their 
timing from an operational view, looking at data flows and logic at a system functional 
view, and looking at the physical interfaces that will enable to required operational 
requirements and system functions. 
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Systems View 
Seven Views 


Framework 

Product 

Product Name 

General Description 

SV-1 

Systems Interface 

Description 

Identification of systems nodes, systems, items, and 
their interconnections within and between nodes 

SV-2 

Description of Systems 
Communications 

Systems nodes, systems, and system items and 
their related communications lay-downs 

SV-3 

Systems-Systems 

Matrix 

Relationships among systems, system-type interfaces 

Planned vs. existing interfaces 

SV-4 

Systems Functionality 

Description 

Systems Functions and data flow among them 

SV-5 

Operational Activity to 

Systems Function 
Traceability Matrix 

Mapping systems to capabilities, functions and 
Operational activities 

SV-6 

Systems Data 
Exchange Matrix 

Systems data elements exchanged between systems 
and the attributes of that exchange 

SV-7 

Systems Performance 
Parameters Matrix 

Performance characteristics of elements for a 
given timeframes) 


Figure 4. System Views one through seven [From: Habayeb, 2005] 


Systems View 
Six Views 


Framework 

Product 

View Name 

General Description 

SV-8 

Systems Evolution 
Description 

Planned migration of systems to more efficient suite, or 
Evolving a current system to future implementation 

SV-9 

Systems Technology 
Forecast 

Time table of emerging software & hardware products 
Technologies and that will impact future architecture 

SV-lOa 

Systems Rules 

Model 

Describes systems functionality constraints due 
to some aspect of systems design or implementation 

SV-10b 

Systems State 
Transition Description 

Describes systems functionality: identifies responses of 
a system to events 

SV-IOc 

Systems Events 

Trace Description 

Describes systems functionality: identifies system changes 
due to critical sequences of events of the Operational View 

SV-11 

Physical Schema 

Physical implementation of the Logical Data Model 
entities: message formats, file structure, physical schema 


Figure 5. System Views eight through eleven [From: Habayeb, 2005] 
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Figure 6. Using Architecture in System Engineering [From: Dickerson and Soules, 

2002] 

Interoperability and integration are implemented by interfaces between systems 
and subsystems. Interfaces provide for functional and physical integration to enable an 
operational capability and as such are critical parts of a system or SoS. Functional and 
physical interfaces drive architecture. A careful analysis of the architecture and system 
engineering views will identify where critical technologies will exist. 

4. DoD System of Systems and Family of Systems Examples 

DoD is currently developing a number of SoSs and FoSs. A selection of four 
legacy, new and mixed FoS and SoSs are reviewed to provide a diversity of SoS and FoS 
perspectives. 
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a. 


Army’s Future Combat System 


The Army is reorganizing its current forces into modular brigade combat 
teams. The Future Force is designed to be a deployable and responsive force and enables 
the Army to move away from the large division-centric structure of the past. Each 
brigade combat team is expected to be highly survivable and the most lethal brigade¬ 
sized unit the Anny has ever fielded. The Army's Future Combat Systems (FCS) is the 
answer to this need and the FCS family of weapons (systems) includes 18 types of 
manned and unmanned ground vehicles, air vehicles, sensors, and munitions li nk ed by a 
information network plus the soldier (note: first deployment of FCS is now 14 plus a 
network and the soldier). The network allows the FCS Family-of-Systems (FoS) to 
operate as a cohesive SoS where the whole of its capabilities is greater than the sum of its 
parts (Future Combat System Program Office, 2007). See Figure 8 for the Operation 
View -1 of FCS. 

FCS has a SoS Common Operating Environment (SOSCOE) central to the 
implementation of the FCS network, which supports multiple mission-critical 
applications independently and simultaneously. It is configurable so that any specific 
instantiation can incorporate only the components that are needed for that instantiation. 
SOSCOE enables straightforward integration of separate software packages, independent 
of their location, connectivity mechanism and the technology used to develop them. 

b. DoD’s Global Combat Support System 

DoD’s Global Combat Support System (GCSS) FoS includes a mix of 
systems that can be tailored to provide focused logistics capabilities. The GCSS FoS 
consists of Service and Defense Agency authoritative single, end-to-end capability 
enabled by infonnation systems from which actionable, real time, accurate data can be 
accessed to manage and monitor units, personnel and equipment through all stages of the 
mobilization process. It is developed and maintained with standard core information 
technology services and capabilities required across the FoS including the Defense 
Information Infrastructure Common Operating Environment (DII COE). GCSS has 
developed a trusted partner certification (TPC) relationship with developers, contractors 
and government agencies for rapid acceptance and distribution of software patches and 
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upgrades in order to maintain the GCSS FOS as current, useful, and up to date on a 
worldwide basis (Joint Chiefs of Staff, 2002). The DII COE is a framework for the 
construction of modular, scalable, distributed Command, Control, Computer, 
Communications, Intelligence, Surveillance and Reconnaissance (C4ISR) computer 
systems. It is a collection of tools for the creation of these systems; it is a set of software 
modules that can be (re-)used to construct these systems (Frazier, 2001) DII COE 
includes a kernel (operating system, security, and software install tools, infrastructure 
services (data exchange, network management, communications) and common support 
applications (e.g., alerts, messaging). 

The GCSS FoS consists of the following systems and their components: 

(1) GCSS Air Force 

(2) GCSS Army 

(3) GCSS Marine Corps 

(4) Navy GCSS capabilities/GCSS maritime 

(5) Global Transportation Network 21 

(6) Joint Total Asset Visibility/DLA Business System Modernization 

(7) Defense Integrated Military Human Resources System 

(8) Theater Medical Information Program 

(9) Defense Infonnation Systems Agency’s (DISA) GCSS (combatant 
commander/JTF) 

(10) Defense Finance and Accounting System Integrated Data 
Environment 
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FCS SoS System Interface Description (OV-1) 
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Figure 7. Future Combat System OV-1 [From: Powell, 2006] 


GCSS Air Force has grown to encompass IT Enterprise Services via 
Service Oriented Architecture approach. See Figure 9 and 10. The Army and Marine 
Corps systems have stayed with a focus on warfighting logistics support. 
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Figure 8. GCSS-AF Capability Evolution [From: GCCS-AF Team, 2006] 
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The Air Force’s Theater Battle Management Command System (TBMCS) 
is the set of application tools used by the Joint Forces Air Component Commander 
(JFACC) to plan and execute theater air operations. The TBMCS is the umbrella 
program for the various systems in an Air Operational Control (AOC) Center. TBMCS 
purpose is to provide a set of connected applications to collect, process and distribute 
data to support employment of air power. This includes the Contingency Theater 
Automated Planning System (CTAPS), Combat Intelligence System (CIS), Wing 
Command and Control System (WCCS), and the Command and Control Information 
Processing System (C2IPS) software applications. It has been deployed in spirals and 
with each spiral has moved towards an Enterprise system giving operators real-time 
access to the status of air operations across the theater via the web. 



GCSS-AF Serves the Warfighter 24/ 7 

Key Operational Metrics 


■ -875,000 registered users 

■ -325,000 Weekly Unique Users 

■ -350,000 PKE users 

■ Worldwide Performance: 2-4s 

■ >600K logins/ week 

■ 2.5M-3.5M pages served daily 

■ Portal availability: >99.8% 

■ >40 production enterprise services, 
aligned with NCES 

■ >200 applications available 

■ Nl PR and SI PR instances (w/ secure 
I nternet access) 

■ Multi-Site w/ COOP Services 

■ SSO (userid/ password and PKE) 

■ High-velocity capability deployment 

■ Over 300 releases annually 

Integrity - Service - Excellence 5 



Figure 9. GCSS Key Operational Metrics [From: GCCS-AF Team, 2006] 

c. Air Force’s Theater Battle Management Command System 

The AOC was known as a ‘System of Systems’ (SoS). As such, it was 
envisioned as a system assembled of other systems so as to offer the capabilities needed 
to perfonn roles assigned to an AOC. Implicit in this was the expectation that the 
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systems from which the AOC was assembled could be composed into an AOC SoS. The 
AOC today is assembled from 80+ applications and systems. There are infrastructure 
elements, communication elements, applications, servers, and databases. The goal was to 
compose the desired capabilities from the elements found in, or which could be brought 
into, the AOC (See Figure 11) (Norman and Kuras, 2004). 


An AOC 



MITRE 2004 C 505* rt» WTS6 Cvrn«n ks rxr* rmm-.iz 

Figure 10. Air Operations Center (AOC) [From: Norman and Kuras, 2004] 

d. DoD Single Integrated Air Picture System of Systems and Army’s 
Integrated Air Missile Defense System of Systems 

DoD has embarked on developing a SoS a Single Integrated Air Picture 
(SIAP) (see OV-1 in Figure 12). SIAP is built via an Integrated Architecture Behavioral 
Model (IABM) which when instantiated in a combat system provides for distributed 
common processing of data/information (see Figure 13). SIAP is an enabling capability 
for mission capabilities such as missile defense. The IABM is built using a Model 
Driven Architecture™ approach. Its goal is a fused, common, continuous, unambiguous 
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track of all airborne objects with one, and only one, track number. The SIAP will fuse 
near-real-time and real-time data allowing users to have identical information about each 
detected airborne object. MDA™ allows developers to focus on specifying platform 
independent business logic and automates the translation of that business logic to target 
programming language, operating system, middleware, database or other information 
technology specifics. 
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Figure 11. Single Integrated Air Picture OV-1 [From: Wilson, 2004] 


The Army’s Integrated Air Missile Defense (IAMD) System of System 
includes SIAP in addition to its missile defense capabilities (see Figure 14). IAMD SoS 
enables a larger defended area against a number of different types of threats while 
providing for flexibility in type of interceptors or other types of weapons. IAMD is 
accomplished via a Common IAMD Battle Command System (IBCS) and plug and fight 
modules at each of the sense, control, engage nodes. The SIAP IABM is part of the plug 
and fight modules. The Army adds service specific common functionality to the plug and 
fight modules. This SoS shows how one SoS can be part of another SoS without 
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overlapping of systems - one SoS is not inside other; SIAP SoS is not contained in IAMD 
SoS. The FCS SOSCOE is to be used with the IAMD SoS in a future spiral. If this does 
happen there will be an IAMD SoS using a FCS SOSCOE with the SIAP SoS IABM. 
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Figure 12. Integrated Architecture Behavior Model [After Ref Wilson, 2004] 



Figure 13. Army’s Integrated Air Missile Defense System of Systems [From: IAMD 
Program Office, 2007] 
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C. INTEROPERABILITY 

There is a continuum of interoperability from exchange of information in non-real 
time thru exchange of raw or semi-processed data as a stream that’s being used and 
manipulated by multiple systems simultaneously. The nature of a mission or task gives 
rise to the type or degree of interoperability required between systems. With the move 
from system-centric to network-centric force operations, agreed to definitions for degrees 
of interoperability is key to the successful development and deployment of a specified 
network-centric capability. 

A taxonomy of degrees of interoperability is useful in identifying critical 
functions and the technologies required to enable interoperability functions. These 
degrees of interoperability are pertinent to the definition of a relevant environment as 
well as definition of the type of system required. 

A review of DoD, industry, and academic literature finds multiple approaches to 
defining the degree of interoperability. The literature review will encompass differing 
types of communication based on tasks or missions to be accomplished as a basis for 
defining degrees of interoperability. 

DoD related literature is useful given the context of joint warfighting 
accomplished via connected systems. Academic literature contains advanced concepts 
and technologies not yet applied that may be useful in the future. The commercial 
industry literature provides the perspective of interoperability via standards that are 
generally required for a profitable business. 

Interoperability is concerned with connectivity, capacity, consistency, bandwidth 
usage, data latency, syntactic compatibility, and undesirable semantic emergent behavior 
(DiMario, 2006). Interoperability is the context in which a SoS definition will be 
defined. An appropriate SoS definition will define the proper operational relevant 
environment in which a TRA is conducted to assess the ability and maturity of a 
technology to support the degree of interoperability required. 


29 



1. DoD/Government Literature Summary 

The Joint Publication 1, ‘Doctrine for the Armed Forces of the United States’ 
states that unified action demands maximum interoperability - The forces, units, and 
systems of all Services must operate together effectively (Joint Chiefs of Staff, 2007) and 
that interoperability should be achieved primarily by a commonality of equipment, 
software, and systems both horizontally and vertically (Joint Chiefs of Staff, 2006). This 
effectiveness is enabled in part through the use of joint and/or interoperable 
communications and infonnation systems that are developed based on a capability- 
focused, effects-based approach to advance Information Technology (IT) and National 
Security Systems (NSS) interoperability (Assistant Secretary of Defense for Networks 
and Information Integration/Department of Defense Chief Information Officer, 2004). 
DoD established a KPP for IT systems. The Interoperability KPP that was originally 
defined has been replaced by the Net-Ready (NR) KPP. The NR KPP is used to assess 
net-ready attributes required for both the technical exchange of information and the end- 
to-end operational effectiveness of that exchange. Meeting the NR KPP was established 
to assure coherent behavior of the interconnected systems to accomplish a common 
mission or task. 

The DoD definitions of interoperability are: 

l.The ability to operate in synergy in the execution of assigned tasks. 2. 

The condition achieved among communications-electronics systems or 
items of communications-electronics equipment when information or 
services can be exchanged directly and satisfactorily between them and/or 
their users. The degree of interoperability should be defined when 
referring to specific cases (Joint Chiefs of Staff, 2001 as amended through 
2007). 

A mental model for network-centric operations and interoperability has been 
proposed by Alberts et al. (see Figure 15). He proposes the following independent 
technical performance metrics to characterize interoperability (Alberts et al., 2001): 

• Completeness (are all the relevant items available, including entities, their 
attributes, and relationships between them) 

• Correctness (are all the items in the system faithful representations of the 
realities they describe) 

• Currency (age of the items of information, often tenned their latency) 
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• Accuracy or Level of Precision (which is conditional on the purpose the 
user has in mind) 

• Consistency across different command centers, functionally specialized 
arenas, and applications 

The model shows the system functions of collection/analysis (sense), decision 
making/C2 (control) and execution (engage) overlaid with the idea of sharing and 
collaboration to accomplish these functions within a networked force. Sharing is 
accomplished today via tactical data li nk s. Collaboration to enable synchronization of 
systems for flexible engagements with an emphasis staying inside the enemies’ 
engagement timeline is the promise of network-centric operations. 


Networking the Force 
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Decisionmaking C2 
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Figure 14. Networking the Force Mental Model [From: Alberts et ah, 2001] 


The collaborative functions that are required to enable synchronization are the 
following (Alberts et ah, 2001): 

• Inclusive: all the relevant actors are involved 

• Collaboration across organizational, functional, spatial, and temporal 
boundaries, including echelons of command 

• Multi-connected (every actor has access to all other actors) 

• Unrestricted communication (between the collaborators) 

• Participatory (all relevant actors are engaged in the process) 

• Continuous (actors are engaged without disruption) 

• Simultaneous (synchronous) 
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• Media-rich (face-to-face, with shared images, information, and data) 

• Domain-rich (involves both the cognitive and the infonnation domains) 

• Content-rich (involves data, infonnation, knowledge, and understandings) 

Interoperability enables Force multiplication and is accomplished through 

common command, control, communications, computers, intelligence, surveillance and 
reconnaissance (C4ISR) specifications, integrated functionality, universal data models 
and other means to enable information sharing and data combining/fusion (Christiansen, 
2005). The DoD mandate for network-centricity as developed and fielded in each system 
empowers users with the ability to easily discover, access, integrate, correlate and fuse 
data/information that support their mission objectives unconstrained by geospatial 
location or time of day (Zavin, 2005). Figure 16 shows the ideal interoperability to 
realize which enables the shortest successful engagement timelines for the lowest cost. 

Not all missions require such tightly coupled operations or collaboration. The 
mission and specific task within a mission will drive the degree of interoperability. The 
mission/task can be accomplished through the allocation and aggregation of individuals, 
organizations, systems, infrastructure, and processes to create and share the data, 
information, and knowledge needed to plan, execute, and assess joint force operations 
and to enable a commander to make decisions better and faster than the adversary (United 
States Joint Forces Command, 2004). 

The Department of Navy (DoN) may have the most challenging interoperability 
problem given the geospatial span of their systems across space, air, surface, land, and 
subsurface. The Navy typically develops and fields multi-mission systems which use 
common planning, C4ISR and weapon system components to conduct multiple missions 
simultaneously. Enabling collaboration across major missions with multiple platforms 
allows the DoN to conserve resources and maximize force multiplication. In addition 
Navy has the challenge of being interoperable with the other Services; the ASN(RDA) 
CHENG depicts the joint interoperability challenge of Navy with the other Services as 
seen in Figure 17. 
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Figure 15. The Military as a Network-Centric Enterprise [From: Alberts et ah, 1999] 

Given that the warfighter requires interoperability and mission and specific task 
within a mission drives the degree of interoperability, all systems may not need to be 
tightly coupled. An interoperability distinction among DoD systems being loosely 
coupled and tightly coupled would need to be made. A tightly coupled set of systems 
would be characterized by interfaces that provide for data and information sharing in the 
interest of cooperation and collaboration usually in near-real time and real-time. An 
example is the Army’s Future Combat System that provides for a networked set of 
systems that in cooperation and collaboration are able to shorten timelines for sense, 
control and engagement of the enemy (Future Combat System Program Office, 2007). A 
loosely coupled set of systems would be characterized by interfaces that provide for data 
and information sharing in the interest of coordination in non-real time or near real-time. 
Examples are the various tactical data li nk s that pass information about an entity’s 
location and message traffic that provides mission planning information and other non- 
real time data and information regarding daily events and affairs. 
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Figure 16. Department of Navy Interoperability Challenge [From: Siel, 2006] 


2. Non DoD Industry Literature Summary 

In the commercial world, Information Technology (IT) businesses are best built 
using agreed to standards whether databases, telecommunications, or computer operating 
systems. An applicable example is the set of standards for the internet called Request for 
Comments (RFC) held in a repository maintained by the Internet Engineering Task Force 
(IETF) Secretariat. The IETF is a large open international community of network 
designers, operators, vendors, and researchers concerned with the evolution of the 
Internet architecture and the smooth operation of the Internet (IETF, 2007). RFCs are the 
community agreed to standardization of protocols and procedures on networking that 
began in 1969 as part of the original Advance Research Program Agency wide-area 
networking (ARPANET) project. The RFC standards are related to the Open Systems 
Interconnection Initiative (OSI) 7 layer model for networking established by the 
International Standards Organization (ISO). The OSI model provides for ‘services’ that 
are a layered, abstract description for communications and computer network protocol 
design. The layers are defined as follows (Subramanian, 2000): 
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Application (Layer 7): This layer supports application and end-user processes. 
Communication partners are identified, quality of service is identified, user authentication 
and privacy are considered, and any constraints on data syntax are identified. Everything 
at this layer is application-specific. This layer provides application services for file 
transfers, e-mail, and other network software services. Telnet and FTP are applications 
that exist entirely in the application level. Tiered application architectures are part of this 
layer. 

Presentation (Layer 6): This layer provides independence from differences in 
data representation (e.g., encryption) by translating from application to network format, 
and vice versa. The presentation layer works to transfonn data into the form that the 
application layer can accept. This layer formats and encrypts data to be sent across a 
network, providing freedom from compatibility problems. It is sometimes called the 
syntax layer. 

Session (Layer 5): This layer establishes, manages and terminates connections 
between applications. The session layer sets up, coordinates, and terminates 
conversations, exchanges, and dialogues between the applications at each end. It deals 
with session and connection coordination. 

Transport (Layer 4): This layer provides transparent transfer of data between end 
systems, or hosts, and is responsible for end-to-end error recovery and flow control. It 
ensures complete data transfer. 

Network (Layer 3): This layer provides switching and routing technologies, 
creating logical paths, kn own as virtual circuits, for transmitting data from node to node. 
Routing and forwarding are functions of this layer, as well as addressing, 
internetworking, error handling, congestion control and packet sequencing. 

Data Link (Layer 2): At this layer, data packets are encoded and decoded into 
bits. It furnishes transmission protocol knowledge and management and handles errors in 
the physical layer, flow control and frame synchronization. The data link layer is divided 
into two sub-layers: The Media Access Control (MAC) layer and the Logical Link 
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Control (LLC) layer. The MAC sublayer controls how a computer on the network gains 
access to the data and pennission to transmit it. The LLC layer controls frame 
synchronization, flow control and error checking. 

Physical (Layer 1): This layer conveys the bit stream — electrical impulse, light or 
radio signal — through the network at the electrical and mechanical level. It provides the 
hardware means of sending and receiving data on a carrier, including defining cables, 
cards and physical aspects. Fast Ethernet, RS232, and ATM are protocols with physical 
layer components. 

3. Academia Literature Summary 

The Carnegie Mellon’s Software Engineering Institute (SEI) is a federally funded 
research and development center conducting software engineering research in software 
system acquisition, architecture, products and interoperability (Carnegie Mellon Software 
Engineering Institute, 2007). SEI has an extensive partner network that stretches beyond 
DoD and for this thesis and literature review is treated as an academic organization that 
works extensively with other academic and commercial industry partners in addition to 
DoD. 

SEI defines interoperability as: 

The ability of a collection of communicating entities to (a) share specified 
information and (b) operate on that information according to a shared 
operational semantics in order to achieve a specified purpose in a given 
context.” SEI cites DoD references for definitions of operational 
interoperability “the ability of systems, units, or forces to provide services 
to and accept services from other systems, units, or forces and to use the 
services so exchanged to enable them to operate effectively together 
(Kasunic and Anderson, 2004). 

The Software Engineering Institute developed a model called Levels of 

Information Systems Interoperability (LISI) (See Figure 18). It was used for a while as a 

representative model for DoD Interoperability KPP. The SEI’s LISI model proposes a 

taxonomy of interoperability: Isolated - non-connected with manual inputs, Connected - 

electronic connection with separate data and applications using homogeneous data 

exchange mechanisms, Functional - minimal common functions with separate data and 

applications using heterogeneous data exchange for basic collaboration, Domain - shared 
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data with separate applications using shared databases, and Enterprise - interactive 
manipulation with shared data and applications using automated distributed information 
exchange applications. The LISI model discriminates among incremental levels of 
information exchange and shared applications (C4ISR Architecture Working Group, 
1997). 


The specific capabilities needed to achieve each level were described in terms of 
four attributes - procedures, applications, infrastructure, and data as follows (C4ISR 
Architecture Working Group, 1997): 

• Procedures: guidance that impact system interoperability, including 
doctrine, mission, architectures, and standards. 

• Applications: functions manifest in the system’s software components, 
from single processes to integrated applications suites. 

• Infrastructure: components that enable interactions between systems, 
including hardware, communications, system services, and security. For 
example, infrastructure considers the protocols, enabling software 
services, and supporting data structures for information flow between 
applications and data. 

• Data: includes the data formats and standards that support interoperability 
at all levels. It embodies the entire range of styles and formats from simple 
text to enterprise data models. 

The literature review regarding interoperability did not reveal an agreed to 
definition or a standard taxonomy of degrees of interoperability. Therefore, a back to 
basics approach was taken to define the types of communication or the purposes of 
communication as a strategy to get to what are the degrees of interoperability. 
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Control (LLC) layer. The MAC sublayer controls how a computer on the network gains 
access to the data and pennission to transmit it. The LLC layer controls frame 
synchronization, flow control and error checking. 

Physical (Layer 1): This layer conveys the bit stream - electrical impulse, light or 
radio signal — through the network at the electrical and mechanical level. It provides the 
hardware means of sending and receiving data on a carrier, including defining cables, 
cards and physical aspects. Fast Ethernet, RS232, and ATM are protocols with physical 
layer components. 
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The Carnegie Mellon’s Software Engineering Institute (SEI) is a federally funded 
research and development center conducting software engineering research in software 
system acquisition, architecture, products and interoperability (Carnegie Mellon Software 
Engineering Institute, 2007). SEI has an extensive partner network that stretches beyond 
DoD and for this thesis and literature review is treated as an academic organization that 
works extensively with other academic and commercial industry partners in addition to 
DoD. 

SEI defines interoperability as: 

The ability of a collection of communicating entities to (a) share specified 
information and (b) operate on that information according to a shared 
operational semantics in order to achieve a specified purpose in a given 
context.” SEI cites DoD references for definitions of operational 
interoperability “the ability of systems, units, or forces to provide services 
to and accept services from other systems, units, or forces and to use the 
services so exchanged to enable them to operate effectively together 
(Kasunic and Anderson, 2004). 

The Software Engineering Institute developed a model called Levels of 

Information Systems Interoperability (LISI) (See Figure 18). It was used for a while as a 

representative model for DoD Interoperability KPP. The SEI’s LISI model proposes a 

taxonomy of interoperability: Isolated - non-connected with manual inputs, Connected - 

electronic connection with separate data and applications using homogeneous data 

exchange mechanisms, Functional - minimal common functions with separate data and 

applications using heterogeneous data exchange for basic collaboration, Domain - shared 
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Coordinate - to bring into a common action, movement, or condition (Merriam-Webster, 
2007), Cooperate - to act or work with another or others: act together or in compliance 
(Merriam-Webster, 2007), Collaborate - to work jointly with others or together 
(Merriam-Webster, 2007), Direct - to regulate the activities or course of (Merriam- 
Webster, 2007). 

These differing purposes lead to differing types of data, infonnation, and 
knowledge flow as well as differing timings. 

D. TECHNOLOGY READINESS ASSESSMENTS 

What is technology and why and how is it assessed. From Merriam’s-Webster 
online (Merriam-Webster, 2007) dictionary, technology comes from the Greek 
technologia, which is a systematic treatment of an art, from techne art, skill + -o- + -logia 
-logy, date: 1859. Its definition includes: 

...the practical application of knowledge especially in a particular area; a 
capability given by the practical application of knowledge; a manner of 
accomplishing a task especially using technical processes, methods, or 
knowledge. 

Most useful may be the idea of technology as a realization of a specific tool, 
technique or method that may be applied consistently to solve a specified problem or 
create something new. 

NASA’s Technology plan defines technology as follows (Bilbro, 2006): 

Technology is defined as the practical application of knowledge to create 
the capability to do something entirely new or in an entirely new way. 

This can be contrasted to scientific research, which encompasses the 
discovery of new knowledge from which new technology is derived, and 
engineering which uses technology derived from this knowledge to solve 
specific technical problems. 

Technology can rarely be developed to a specified schedule and breakthrough 
technologies are rarely available to support a program schedule. Failure to account for 
the time to develop technology contributes significantly to schedule slip and cost overrun 
for a program. Even if a technology has been fielded in one system doesn’t mean that it 
is mature enough to meet the requirements of another system or meet SoS requirements. 
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Most IT technologies that have been fielded in a system must be modified to work in an 
enterprise or SoS/IT application. Technology assessments are used to identify the 
development activities and risks associated with technology development in support of a 
program. 

The fonnalization of technology assessments within the government was started 
by NASA. John C. Mankins, of NASA, first documented Technology Readiness Levels 
(TRLs) in his white paper ‘Technology Readiness Levels, A White Paper,’ April 6, 1995 
(Mankins, 1995). 

This literature review will include the DoD’s TRA Deskbook, GAO reports, 
NASA technology assessment approaches and Service specific technology assessment 
strategies and initiatives. In addition, personal participation and observation in the 
execution of the DoD’s first Joint SoS TRA will be included. Most of the documented 
information regarding technology assessment is within DoD and NASA; however, this 
review will include personal observations of the non-DoD Industry. In particular, 
guidance that would be pertinent to a SoS TRA is included. 

1. Technology Readiness Assessment Government/Department of 
Defense Industry Literature Review 

The definition of a TRA per the TRA Deskbook: 

...is a systematic, metrics-based process and accompanying report that 
assesses the maturity of certain technologies [(called Critical Technology 
Elements (CTEs)) used in systems.” The TRA report includes “how the 
CTEs are identified, why they are important to the program and an 
independent assessment of their maturity ((DUSD(S&T)), 2005). 

There has been an increased interest by Congress and Office of Secretary of 
Defense on managing programs within cost, schedule and performance. One of the 
contributors to delays and cost overruns is the use of immature technologies. DoD 
Directive Number 5000.1 May 12, 2003, The Defense Acquisition System, USD(AT&L), 
DoD Instruction Number 5000.2 May 12, 2003, Operation of the Defense Acquisition 
System, USD(AT&L) and DoD Acquisition Guidebook, Last Modified on: 12/20/2004 
(USD(AT&L), 2006), section 10.5.2 gives guidance regarding Technology Readiness 
Assessments (TRAs) to support program initiation for ships (usually Milestone A, 
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Milestone B (typical program initiation) and Milestone C (system/product production) 
decisions. Fonnal, independent TRAs approved by a Service S&T Executive and the 
Service Acquisition Executive are required for MDAP and MAIS acquisitions. The latest 
TRA Deskbook dated May 2005, prepared by the Deputy Under Secretary of Defense for 
Science and Technology (DUSD(S&T)) provides the latest instructions and guidance for 
TRAs. TRAs should be conducted for each block or spiral of an acquisition program. 
All of these documents are basically silent on FoS TRAs and include very little guidance 
on SoS TRAs. 

In the early 1990s, DoD adopted NASA’s Technology Readiness Level (TRLs) 
scale (see Figure 19) with minor modifications and developed a TRA Deskbook 
containing guidance and best practices on technology development and assessment using 
these TRLs. Detailed descriptions of the Hardware TRLs can be found in Table 1. 
NASA’s TRLs developed during the 1970’s and 1980’s were primarily applied to 
hardware programs. In the last twenty years, software development has become more 
prevalent. The hardware TRLs have been modified to reflect the aspect of software 
maturity. Both NASA and DoD have developed software TRLs. See Figure 20 for 
NASA’s software TRLs and Table 2. for DoD descriptions of software TRLs. Tenns 
used in these descriptions such as breadboard and high fidelity environment are defined 
just after Table 2. 

Software is mostly associated with IT systems. The TRA Deskbook and Gold and 
Jabubek in their article ‘Technology Readiness Assessments for IT and IT-Enabled 
Systems’ ((DUSD(S&T)), 2005) and (Gold and Jakubak, 2005), define four types of IT 
systems: 

• Business systems - off-the-shelf information system components and 
COTS software assembled together in a new environment to support the 
business and management functions of an organization 

• Net-reliant (battle management) systems - typically command and control; 
battle management systems; or intelligence, surveillance, and 
reconnaissance systems. The net-reliant system is characterized by an 
intense real-time requirement 
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• Network infrastructure (or services provider) - backbone and services 
systems for network management, management of large databases and 
glue logic to execute and retrieve services across a Wide Area Network of 
varying security levels. 

• Embedded systems - functionality is enabled by IT but not driven by IT 
itself. Embedded systems emphasize using computer hardware and 
software to automate internal functions of a weapon system such as 
platform control and status, sensor signal and data processing, and 
weapons tasking. 

Over time DoD has developed TRLs for other categories such as manufacturing 
and medical. Details of the DoD manufacturing and medical TRLs can be found in the 
TRA Deskbook. 

Assessment begins by identifying the operational environment and the KPPs and 
other required capabilities for a system. As the system is engineered those technologies 
that enable the meeting of operational requirements in the environment that the system 
will be employed, support manufacturing of hardware or development of software (e.g., 
Integrated Development Environment) are identified. Key drivers of the operational 
environment must be identified, so that demonstration and test results of a CTE are 
analyzed with respect to these drivers. CTEs are assessed using the appropriate TRLs. 
CTEs include hardware, software, algorithms, techniques, and methods. Assessments are 
conducted throughout system acquisition. The following applies per the TRA Deskbook. 

For a technology to be critical, the answer to one of the following questions must 
be ‘yes’: 

• Does the technology directly impact an operational requirement? 

• Does the technology have a significant impact on an improved delivery 
schedule? 

• Does the technology have a significant effect on the system’s 
affordability? 

• If this is a spiral development, is the technology essential to meet the 
spiral deliverables? 

In addition, the answer to one of the following questions must also be ‘yes’: 

• Is the technology new or novel? 

• Is the technology modified? 
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• Has the technology been repackaged so that a new relevant environment is 
realized? 

• Is the technology expected to operate in an environment and/or achieve a 
performance beyond its original design intention or demonstrated 
capability? 

Identification of the operational relevant environments will like include the 
following ((DUSD(S&T)), 2005): 

• Physical Environment. Including but not limited to: mechanical 
components, processors, servers, and electronics; kinetic and kinematic; 
thermal and heat transfer; electrical and electromagnetic; climatic— 
weather, temperature, particulate; network infrastructure 

• Logical Environment. Including but not limited to: software (algorithm) 
interfaces; security interfaces; Web-enablement 

• Data Environment. Including but not limited to: data formats and 
databases; anticipated data rates, data delay and data throughput; data 
packaging and framing 

• Security Environment. Including but not limited to: connection to 
firewalls; security appliques; rates and methods of attack 

• User and Use Environment. Including but not limited to: scalability; 
upgradability; user behavior adjustments; user interfaces; organizational 
change/realignments with system impacts; implementation plan. 


System Test. Launch 
& Operations 


System Subsystem 
Development 



TRL 9 
TRL 8 

TRL 7 


Technology 

Demonstration 


Technology 

Development 


TRL 6 
TRL 5 


TRL 4 


Research to Prove 
Feasibility 


TRL 3 


Actual system ‘flight proven” through successful 
mission operations 

Actual system completed and “flight qualified” through 
test and demonstration 

System prototype demonstration in a operational 
environment 

System subsystem model or prototype demonstration 
in a relevant environment 

Component and/or breadboard validation in relevant 
environment 

Component and/or breadboard validation in laboratory 
environment 

Analytical and experimental critical function and/or 
characteristic proof-of-concept 


Basic Technology 
Research 


Y TRL2 ^ 
TRL 1 


L. A 


Technology concept and/or application formulated 


Basic principles observed and reported 


Figure 18. Hardware Technology Readiness Level (TRL) scale [From: National 
Aeronautics and Space Administration, 2007] 
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H/W 

TRL 

DEFINITION 

DESCRIPTION 

SUPPORT INFORMATION 

1 

Basic principles 
observed and 
reported. 

Lowest level of technology readiness. 

Scientific research begins 
to be translated into applied research and 
development (R&D). Examples might 
include paper studies of a technology’s basic 
properties. 

Published research that identifies 
the principles that underlie this 
technology. References to who, 
where, when. 

2 

Technology 
concept and/or 
application 
formulated. 

Invention begins. Once basic principles are 
observed, practical applications can be 
invented. Applications are speculative, and 
there may be no proof or detailed analysis to 
support the assumptions. Examples are 
limited to analytic studies. Once basic 
principles are observed, practical 
applications can be invented. Applications 
are speculative, and there may be no proof or 
detailed analysis to support the assumptions. 
Examples are limited to analytic studies 
using synthetic data 

Publications or other references 
that outline the application being 
considered and that provide 
analysis to support the concept. 
Applied research activities, 
analytic studies, small code units, 
and papers comparing competing 
technologies. 

3 

Analytical and 
experimental 
critical function 
and/or 

characteristic 
proof of 
concept. 

Active R&D is initiated. This includes 
analytical studies and laboratory studies to 
physically validate the analytical predictions 
of separate elements of the technology. 
Examples include components that are not 
yet integrated or representative. 

Results of laboratory tests 
performed to measure parameters 
of interest and comparison to 
analytical predictions for critical 
subsystems. References to who, 
where, and when these tests and 
comparisons were performed. 

4 

Module and/or 

subsystem 

validation in a 

laboratory 

environment 

(i.e., software 

prototype 

development 

environment). 

Basic technological components are 
integrated to establish that they will work 
together. This is relatively “low fidelity” 
compared with the eventual system. 

Examples include integration of “ad hoc” 
hardware in the laboratory. 

System concepts that have been 
considered and results from testing 
laboratory scale breadboard(s). 
References to who did this work 
and when. Provide an estimate of 
how breadboard hardware and test 
results differ from the expected 
system goals. 

5 

Module and/or 
subsystem 
validation in a 
relevant 
environment. 

Fidelity of breadboard technology increases 
significantly. The basic technological 
components are integrated with reasonably 
realistic supporting elements so they can be 
tested in a simulated environment. Examples 
include “high fidelity” laboratory integration 
of components. 

Results from testing a laboratory 
breadboard system are integrated 
with other supporting elements in 
a simulated operational 
environment. How does the 
“relevant environment” differ 
from the expected operational 
environment? How do the test 
results compare with 
expectations? What problems, if 
any, were encountered? Was the 
breadboard system refined to more 
nearly match the expected system 
goals? 
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H/W 

TRL 

DEFINITION 

DESCRIPTION 

SUPPORT INFORMATION 

6 

Module and/or 
subsystem 
validation in a 
relevant end-to- 
end 

environment. 

Representative model or prototype system, 
which is well beyond that of TRL 5, is tested 
in a relevant environment. Represents a 
major step up in a technology’s demonstrated 
readiness. Examples include testing a 
prototype in a high fidelity laboratory 
environment or in a simulated operational 
environment. 

Results from laboratory testing of 
a prototype system that is near the 
desired configuration in terms of 
performance, weight, and volume. 
How did the test environment 
differ from the operational 
environment? Who performed the 
tests? How did the test compare 
with expectations? What 
problems, if any, were 
encountered? What are/were the 
plans, options, or actions to 
resolve problems before moving to 
the next level? 

7 

System 
prototype 
demonstration 
in an 

operational 

high-fidelity 

environment. 

Prototype near or at planned operational 
system. Represents a major step up from 

TRL 6 by requiring demonstration of an 
actual system prototype in an operational 
environment (e.g., in an aircraft, in a vehicle, 
or in space). Examples include testing the 
prototype in a test bed aircraft. 

Results from testing a prototype 
system in an operational 
environment. Who performed the 
tests? How did the test compare 
with expectations? What 
problems, if any, were 
encountered? What are/were the 
plans, options, or actions to 
resolve problems before moving to 
the next level? 

8 

Actual system 
completed and 
mission 
qualified 
through test 
and 

demonstration 
in an 

operational 

environment. 

Technology has been proven to work in its 
final form and under expected conditions. In 
almost all cases, this TRL represents the end 
of true system development. Examples 
include developmental test and evaluation of 
the system in its intended weapon system to 
determine if it meets design specifications. 

Results of testing the system in its 
final configuration under the 
expected range of environmental 
conditions in which it will be 
expected to operate. Assessment 
of whether it will meet its 
operational requirements. What 
problems, if any, were 
encountered? What are/ were the 
plans, options, or actions to 
resolve problems before finalizing 
the design? 

9 

Actual system 

proven through 

successful 

mission- 

proven 

operational 

capabilities. 

Actual application of the technology in its 
final form and under mission conditions, 
such as those encountered in operational test 
and evaluation (OT&E). Examples include 
using the system under operational mission 
conditions. 

OT&E reports. 


Table 1. Hardware Technology Readiness Level Descriptions [From: (DUSD(S&T)), 
2005] 
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Technology Readiness Levels 

Applied to Software 

(v5 6/21/99 ARC/GSFC) 



System Test, 
Launch & 
Operations 


System/Subsystem 

Development 



TRL 7 


Technology 

Demonstration 


Technology 

Development 


Research to 
Prove Feasibility 


Basic Technology 
Research 



TRL 9: Actual system “mission proven” through successful mission operations 

Thoroughly debugged software readily repeatable. Fully integrated with operational hardware/software 
systems. All documentation completed. Successful operational experience. Sustaining software 
engineering support in place. Actual system fully demonstrated. 

TRL 8: Actual system completed and “mission qualified” through test and 
demonstration in an operational environment Thoroughly debugged software. Fully 
integrated with operational hardware and software systems. Most user documentation, training 
documentation, and maintenance documentation completed. All functionality tested in simulated and 
operational scenarios. V&V completed. 

TRL 7: System prototype demonstration in high-fidelity environment (parallel or 
shadow mode operation) Most functionality available for demonstration and test. Well integrated 
with operational hardware/software systems. Most software bugs removed. Limited documentation 
available. 

TRL 6: System/subsystem prototype demonstration in a relevant end-to-end 
environment Prototype implementations on full scale realistic problems. Partially integrated with 
existing hardware/software systems. Limited documentation available. Engineering feasibility fully 
demonstrated. 

TRL 5: Module and/or subsystem validation in relevant environment Prototype 
implementations conform to target environment / interfaces. Experiments with realistic problems. 

Simulated interfaces to existing systems. 

TRL 4: Module and/or subsystem validation in laboratory environment Standalone 
prototype implementations. Experiments with full scale problems or data sets. 

TRL 3: Analytical and experimental critical function and/or characteristic proof- 
of-concept Limited functionality implementations. Experiments with small representative data sets. 
Scientific feasibility fully demonstrated. 

TRL 2: Technology concept and/or application formulated Basic principles coded. 
Experiments with synthetic data. Mostly applied research. 

TRL 1: Basic principles observed and reported Basic properties of algorithms, 
representations 8 concepts. Mathematical formulations. Mix of basic and applied research. 

Comments to kswanson@mail.arc.nasa.gov 


Figure 19. NASA Software Technology Readiness Levels [From: NASA, 2007] 


s/w 

TRL 

DEFINITION 

DESCRIPTION 

SUPPORT 

INFORMATION 

1 

Basic principles 
observed and 
reported. 

Lowest level of software technology readiness. 

A new software domain is being investigated by 
the basic research community. This level 
extends to the development of basic use, basic 
properties of software architecture, mathematical 
formulations, and general algorithms. 

Basic research activities, 
research articles, peer- 
reviewed white papers, 
point papers, early lab model 
of basic concept may be 
useful for substantiating the 
TRL level. 

2 

Technology 
concept and/or 
application 
formulated. 

Once basic principles are observed, practical 
applications can be invented. Applications are 
speculative, and there may be no proof or 
detailed analysis to support the assumptions. 
Examples are limited to analytic studies using 
synthetic data 

Applied research activities, 
analytic studies, small code 
units, and papers comparing 
competing technologies. 

3 

Analytical and 
experimental 
critical function 
and/or 

characteristic 
proof of concept. 

Active R&D is initiated. The level at which 
scientific feasibility is demonstrated through 
analytical and laboratory studies. This level 
extends to the development of 
limited functionality environments to validate 
critical properties and analytical predictions 
using nonintegrated software components and 
partially representative data. 

Algorithms run on a 
surrogate 

processor in a laboratory 
environment, instrumented 
components operating in 
laboratory environment, 
laboratory results showing 
validation of critical 
properties. 
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S/W 

TRL 

DEFINITION 

DESCRIPTION 

SUPPORT 

INFORMATION 

4 

Module and/or 

subsystem 

validation in a 

laboratory 

environment 

(i.e., software 

prototype 

development 

environment). 

Basic software components are integrated to 
establish that they will work together. They are 
relatively primitive with regard to efficiency and 
robustness compared with the eventual system. 
Architecture development initiated to include 
interoperability, reliability, maintainability, 
extensibility, scalability, and security issues. 
Emulation with current/legacy elements as 
appropriate. Prototypes developed to 
demonstrate different aspects of eventual 
system. 

Advanced technology 
development, stand-alone 
prototype solving a synthetic 
full-scale problem, or 
standalone prototype 
processing fully 
representative data sets. 

5 

Module and/or 
subsystem 
validation in a 
relevant 
environment. 

Level at which software technology is ready to 
start integration with existing systems. The 
prototype implementations conform to target 
environment/interfaces. Experiments with 
realistic problems. Simulated interfaces to 
existing systems. System software architecture 
established. Algorithms run on a processor(s) 
with characteristics expected in the operational 
environment. 

System architecture diagram 
around technology element 
with critical performance 
requirements defined. 
Processor selection analysis, 
Simulation/Stimulation 
(Sim/Stim) Laboratory 
buildup plan. Software 
placed under configuration 
management. COTS/GOTS 
in the system software 
architecture are identified. 

6 

Module and/or 
subsystem 
validation in a 
relevant end-to- 
end 

environment. 

Level at which the engineering feasibility of a 
software technology is demonstrated. This level 
extends to laboratory prototype implementations 
on full-scale realistic problems in which the 
software technology is partially integrated with 
existing hardware/ software systems. 

Results from laboratory 
testing of a prototype 
package that is near the 
desired configuration in 
terms of performance, 
including physical, logical, 
data, and security interfaces. 
Comparisons between tested 
environment and operational 
environment analytically 
understood. Analysis and 
test measurements 
quantifying contribution to 
system-wide requirements 
such as throughput, 
scalability, and reliability. 
Analysis of human- 
computer 

(user environment) begun. 

7 

System 
prototype 
demonstration in 
an operational 
high-fidelity 
environment. 

Level at which the program feasibility of a 
software technology is demonstrated. This level 
extends to operational environment prototype 
implementations where critical technical risk 
functionality is available for demonstration and a 
test in which the software technology is well 
integrated with operational hardware/software 
systems. 

Critical technological 
properties are measured 
against requirements in a 
simulated operational 
environment. 

8 

Actual system 
completed and 
mission 

qualified through 
test and 

Level at which a software technology is fully 
integrated with operational hardware and 
software systems. Software development 
documentation is complete. All functionality 
tested in simulated and operational scenarios. 

Published documentation 
and product technology 
refresh build schedule. 
Software resource reserve 
measured and tracked. 
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s/w 

TRL 

DEFINITION 

DESCRIPTION 

SUPPORT 

INFORMATION 


demonstration in 
an operational 
environment. 



9 

Actual system 
proven through 
successful 
mission- proven 
operational 
capabilities. 

Level at which a software technology is readily 
repeatable and reusable. The software based on 
the technology is fully integrated with 
operational hardware/software systems. All 
software documentation verified. Successful 
operational experience. Sustaining software 
engineering support in place. Actual system. 

Production configuration 
management reports. 
Technology integrated into a 
reuse “wizard”; out-year 
funding established for 
support activity. 


Table 2. DoD Software Technology Readiness Levels [From: (DUSD(S&T)), 2005] 


Definitions applicable to TRL descriptions ((DUSD(S&T)), 2005): 

Breadboard: Integrated components that provide a representation of a 

system/subsystem and which can be used to determine concept feasibility and to develop 
technical data. Typically configured for laboratory use to demonstrate the technical 
principles of immediate interest. May resemble final system/sub system in function only. 

High Fidelity: Addresses form, fit and function. High-fidelity laboratory 
environment would involve testing with equipment that can simulate and validate all 
system specifications within a laboratory setting. 

Low Fidelity: A representative of the component or system that has limited 
ability to provide anything but first order infonnation about the end product. Low-fidelity 
assessments are used to provide trend analysis. 

Model: A functional form of a system, generally reduced in scale, near or at 
operational specification. Models will be sufficiently hardened to allow demonstration of 
the technical and operational capabilities required of the final system. 

Operational Environment: Environment that addresses all of the operational 
requirements and specifications required of the final system to include 
platform/p ackaging. 

Prototype: A physical or virtual model used to evaluate the technical or 
manufacturing feasibility or military utility of a particular technology or process, concept, 
end item or system. 
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Relevant Environment: Testing environment that simulates the key aspects of the 
operational environment. 

Simulated Operational Environment: Either 1) a real environment that can 
simulate all of the operational requirements and specifications required of the final 
system, or 2) a simulated environment that allows for testing of a virtual prototype; used 
in either case to detennine whether a developmental system meets the operational 
requirements and specifications of the final system. 

The TRA Deskbook recommends certain technology readiness by specific 
acquisition milestones to mitigate program risk (see Figure 21). It is required that the 
TRA be conducted by an independent panel of subject matter experts typically assembled 
by the component S&T Executive prior to a MS event. It is required that the TRA be 
conducted by and independent technology panel typically assembled by the component 
S&T Executive prior to a MS event. Assessments are made based on expert judgments 
and the technology design documentation, test results, and program requirements 
supplied by the PM. 

Ideally, during the development of the Initial Capabilities Document (ICD) there 
should be involvement of the S&T community to ensure that materiel elements for the 
needed capabilities are plausible. At the beginning of concept development, alternatives 
are proposed and analyzed. During concept exploration there must be a balancing 
between required capabilities, cost, and availabilities of technologies. CTE technology 
development and/or modification is defined during the concept phase of a system and is 
restrained and constrained by system concepts of employment, Key Performance 
Parameters, System requirements (legacy constraints), and availability of money, time, 
and maturing Science and Technology (S&T) technologies and existing technologies. 
CTE should be assessed with respect to technology maturity and technical risk. Ideally, 
alternatives are based on technologies that typically are at least TRL 4 such that an 
evaluation of the expected performance and cost of a system alternative can be analyzed 
with moderate risk. If the system alternatives can not meet performance with 
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technologies of at least TRL 4, ideally an S&T project is started for a technology area and 
the original program progresses with a lower performance requirement for its initial 
spirals or blocks. 



Process entry at Milestones A, B r or C 
Entrance criteria met before entering phase 
Evolutionary Acquisition or Single Step to Full 
Capability 


Joint Capabilities 
Integration & 
Development 
System (JCIDS) 


A A 


(Program 

Initiation) 


L 


A 


I0C 


Pre-Systems Acquisition 


Systems Acquisition 
-CPD 


FOC 


Concept 

Technology 

System Development 

Production & 

Operations & 

Refinement 

Development 

& Demonstration 

Deployment 

Support 

\ Concept 
✓ Decision 


A 

V Review 

LRIP/IOT&E A DbcIs :n 
yf Review 



Sustainment 


TRAs required at MS B, MS C, aud program 
initiation for ships (usually MS A). 


Figure 20. Recommended Technology Readiness Assessments by Milestone [From: 

Mandelbaum, 2005] 

At the same time, one should identify technology maturation plans and 
demonstration and test requirements required to be accomplished in the Technology 
Demonstration phase of the program and beyond. This information is required to develop 
the Technology Development Strategy (TDS), which is required at MS A. For new 
radars, ships, and other systems that have a long development times or long 
procurement/manufacture/build times and/or manufacturing facilities are required to be 
developed, it is paramount that critical technologies are identified very early during the 
concept stage. These technologies typically take an extraordinary long time to develop 
and require a significant investment. 

In DoD it is common practice to develop or use advanced technologies to keep a 
warfighting edge. Advanced technologies will most likely be used in ways that have 
never been accomplished before. Historically, DoD PMs and their staffs have been too 
optimistic with regard to technology readiness and program schedules. Many programs 
actually progress without an S&T program and develop the technology within their 
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program adding undo risk to their program. This is typically rationalized by the PM in 
order to meet program deadlines; the PM should coordinate and collaborate with the S&T 
community. 

When a program is exiting the Technology Development phase and entering the 
SDD phase, the PM should be sure that the program is affordable and can be developed 
for production in a reasonable amount of time (typically less than five years); the 
statistics are against the PM being able to do this with immature technologies. 

As a program nears Milestone B (~1 year prior), the Program Manager should 
formally propose the list of all CTEs to his respective Service’s S&T Executive for 
approval. These should be easily identifiable from the system engineering artifacts 
related to functional architecture and the physical architecture which shows the system 
design broken down into all its subsystems and components. The program’s work 
breakdown structure should be used (per acquisition guidance) to facilitate the 
enumeration of the CTEs. It is recommended that the list be inclusive of all technologies 
under consideration. For those technologies that have low TRLs, alternative technologies 
with higher TRLs should be included. The Program Manager (PM) should provide a 
draft technology readiness assessment to the Service S&T Executive and is required to 
defend the technology maturity levels claimed. It is the job of the S&T Executive to 
certify the TRA. This is accomplished for both MS B and MS C. Technologies are 
required to be a minimum of TRL6 for MS B and TRL 7 for MS C. There may be 
changes in the complement of technologies during program development. For 
technology assessments with respect to TRL 7, the environment should be the system in 
development. If no data/infonnation exists under an operationally relevant environment, 
these activities must be conducted prior to MS C in order to have the data. It is the 
responsibility of the PM to keep the S&T Executive and TRA current on a regular basis. 

Typical Questions that must be answered during a TRA: 

1. Is this a new technology? Is the technology novel (a mature technology 
being used in a new way? Is the technology being modified or being used 
in a different enviromnent? (Summarize what functions are being 
modified and how)? 

2. Who developed the technology? When? Is it in another program? Is it 
fielded (for how long)? 
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3. What KPP(s) or operational requirements does this technology support? 
(i.e. Completeness, Clarity, Commonality, Net-Ready) 

4. What documents exist that describe the technology and its performance? 
(State documents title and date and provide to panel) 

5. What system and subsystem requirements are satisfied with this 
technology? What other technologies are used in conjunction with this 
technology to meet these requirements? 

6. What functional capabilities does this technology map to? 

7. What trade studies or concept papers exist for this technology element? 
(List name of trade study (date) and provide copy to panel)? 

8. Brief Summary of Development products, architecture and system 
engineering artifacts as required. 

9. Where was testing conducted? What scenarios/threats/ environments were 
used? Are the scenarios certified as representative of the operational 
environment? How were these tailored to test technology components not 
in a development system? What artifacts are available (provide those 
available)? What was the performance with respect to the requirements? 

10. When will development and testing be complete at a subsystem, system, 
and SoS levels. 

The TRL descriptions and supporting information should be tailored with 
program specific details. This allows program personnel to communicate clearly about 
what specific steps are required to mature a technology. See Table 3 for an example. 
The TRA Deskbook provides many good examples of this annotation as well as details 
relevant to assessments of specific types of hardware, software/IT, manufacturing and 
biomedical technologies. 


TRL 

Example Description 

5 

Strategies identified to mitigate technical and cost risk: Change from 2-in. to 4-in. wafers for 
Molecular Beam Epitaxy (MBE) growth. Identify machines to automate bar stacking in coating 
fixtures. Preload bars in bonding fixtures to reduce solder thickness in laser diode array package 
and improve reliability and thermal performance 


Table 3. Example of TRL tailoring [From: (DUSD(S&T)), 2005] 


The following content from the TRA Deskbook is applicable for SoS TRAs: 

• Software CTE identification and assessment typically includes algorithms/ 
techniques/methods, components, subsystem, system/program/package, 
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and SoS elements. The environment description should include 
integration aspects, user environment, logical relationships, data 
environment, and external interfaces. 

• CTEs are most likely found perfonning functions supporting 
synchronization, timeliness, accuracy, dissemination and consistency of 
data requirements, operating system environment, workstations, servers, 
other special processing needs and Quality of Service and throughput of 
networks. 

• IT systems engineering creates a data model that exposes data types and 
their relationships. This data model includes a description of data flow 
(i.e., how the activities of the IT system affect the data) and the 
distribution of computational processes over the system. The data model is 
analogous to the functional architecture for a hardware-centric system. 

• Consider the following during the assessment: obsolescence, scalability, 
data storage, number and type of applications, processor requirements and 
throughput including appropriate hardware components that are required 
to meet these requirements. 

• Specifically identify and assess Commercial-off-the-Shelf (COTS) and 
industry standards and their ability to support military needs (including but 
not limited to reliability, availability, and security) over the long term 
without undo modification and costs. 

• Claiming technical readiness in an operation environment (TRL 7 or 
higher) requires a detailed architecture that fully exposes all components 
and elements affecting the operation of the critical software element. 
Claiming technical readiness in a relevant environment (TRL 6 or higher) 
requires evidence of the acceptable perfonnance of the software element 
under operational factors, including, for example, system loading, user 
interaction, and realistic communications environment (e.g., bandwidth, 
latency, jitter). In other words, claiming a TRL 5 or higher requires a 
detailed architecture, and claiming a TRL 7 or higher requires, in addition 
to the detailed architecture, defining the operational environment and 
evidence of acceptable performance in the operational environment. 

There are a variety of methods used across DoD by the Services to better assess 
technology readiness. Two exemplars are included in this thesis. The Missile Defense 
Agency (MDA) has developed a detailed checklist for each TRL both hardware and 
software. The MDA hardware TRL checklist in included as Table 3. MDA has clarified 
specific steps that must be met to clearly move to the next level of readiness. MDA also 
has a draft Software TRL checklist similar to the Hardware TRL checklist included here. 
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TECHNOLOGY READINESS LEVEL 1 

HARDWARE MATURITY CHECKLIST 

DoD TRL 1 
Definition 

TRL 1 Hardware Maturity Criteria 

Certification 

Authority 

Basic Principles 
Observed and 
Reported. Lowest 
level of technology 
readiness. 

Scientific research 
begins to be 
translated into 
technology’s basic 
properties. 

TRL 1 certification is not dependent on criteria. The Principal 
Investigator determines a TRL 1 with conceptualization. 

Principal 

Investigator. 



TECHNOLOGY READINESS LEVEL 2 

HARDWARE MATURITY CHECKLIST 

DoD TRL 2 
Definition 

TRL 2 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

Technology 

Concept and/or 

Application 

Formulated 

Invention begins. 
Once Basic 
principles are 
observed, practical 
applications can be 
invented. The 
application is 
speculative and 
there is no proof or 
detailed analysis to 
support the 
assumption. 

Examples are still 
limited to paper 
studies. 

1. Basic physical principles have been 
confirmed independently. Physical laws 
and assumptions used in new 
technologies defined. Provide details. 

□ 

□ 

□ 

Principal 
Investigator 
and MDA/DV 
Project 

Sponsor 
.Certifies that 
technology has 
met all 

applicable TRL 

2 Hardware 
Maturity 

Criteria and has 
thus achieved 
TRL 2 status. 

2. Basic elements of technology have been 
identified. Provide a list of the elements 
of technology. Describe and compare 
relationship of old to new elements. 

□ 

□ 

□ 

3. An apparent theoretical or empirical 
design solution identified. Describe 
design in as much detail as possible. 

□ 

□ 

□ 

4. Components of technology have been 
partially characterized. Provide details 
on characterization for each component. 

□ 

□ 

□ 

5. Design techniques/codes have been 
identified/developed and performance 
predictions made for each element. 

Provide details on design 
techniques/code and predictions made. 

□ 

□ 

□ 

6. Potential BMDS application(s) have been 
identified. Provide a discussion of each 
of these applications. 

□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 
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TECHNOLOGY READINESS LEVEL 3 

HARDWARE MATURITY CHECKLIST 

DoD TRL 3 
Definition 

TRL 3 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

Analytical and 
Experimental 
Critical Function 
and/or 

Characteristic 
Proof of Concept. 

Active research 
and development 
is initiated. This 
includes analytical 
studies and 
laboratory studies 
to physically 
validate analytical 
predictions of 
separate elements 
of the technology. 
Examples include 
components that 
are not yet 
integrated or 
representative. 

1. Performance predictions of elements of 
technology capability validated by: 

a. Analytical Studies, 

b. Laboratory Experiments, and/or 

c. Modeling and Simulation 

Provide details of the studies, 

experiments, and M&S. 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Principal 
Investigator and 
MDA/DV 

Project Sponsor 
in consultation 
with potential 

End Users(s). 
Certifies that 
technology has 
met all applicable 
TRL 3 Hardware 
Maturity Criteria 
and has thus 
achieved TRL 3 

status. 

2. Scaling studies have been started. 

Define the goals of the studies and how 
the goals relate to the BMDS mission. 

□ 

□ 

□ 

3. Preliminary performance characteristics 
and measures have been identified and 
estimated. Quantify level of 
performance. 

□ 

□ 

□ 

4. Cross technology effects (if any) have 
begun to be identified. Identify other- 
new or in development technology that 
could increase performance and 
reduce risk. 

□ 

□ 

□ 

5. Design techniques/codes have been 
identified and defined to the point 
where small applications may be 
analyzed/simulated. Provide details. 

□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 


TECHNOLOGY READINESS LEVEL 4 

HARDWARE MATURITY CHECKLIST 

DoD TRL 4 
Definition 

TRL 4 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certificati 

on 

Authority 

Component and/or 
Breadboard 
Validation in 
Laboratory 
Environment. 

Basic technological 
components are 
integrated to 
establish that the 
pieces will work 

1. Low fidelity hardware technology “system” 
integration and engineering completed in a 
lab environment with hardware in the 
loop/computer in the loop tools to establish 
component compatibility. Provide summary 
reports of efforts including results. 

□ 

□ 

□ 

Deputy for 

DV in 

consultatio 

n with 

MDA/DV 

Project 

Sponsor, 

SE,and 

potential 

End 

2. Technology demonstrates basic functionality 
in simplified environment. Describe 
demonstrated functionality and provide 
summary of collected data. 

□ 

□ 

□ 
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TECHNOLOGY READINESS LEVEL 4 

HARDWARE MATURITY CHECKLIST 

DoD TRL 4 
Definition 

TRL 4 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certificati 

on 

Authority 

together. This is 
relatively "low 
fidelity" compared 
to the eventual 
system. Examples 
include integration 
of "ad hoc" 
hardware in a 
laboratory. 

3. Scaling studies have continued to next higher 
assembly from previous assessment, 
a. Scaling documents and diagrams of 
technology have been completed, 
b. Scalable technology prototypes have 
been produced. 

BMDS mission enhancement(s) clearly 
defined within goals of the study. 

□ 

□ 

□ 

□ 

□ 

□ 

Users(s); 

ATC 

Secretariat 

informed. 

Certifies 

that 

technology 
has met all 
applicable 
TRL 4 
Hardware 
Maturity 
Criteria and 
has thus 
achieved 

TRL 4 

status. 

4. Integration studies have been started. 

Provide a ROM integration cost estimate, 
with Systems Engineering, System Executive 
Officer, and end user inputs and 
coordination. 

□ 

□ 

□ 

5. Draft conceptual hardware and software 
designs have been documented. Provide 
copy of documentation. 

□ 

□ 

□ 

6. Some software components are available. 
Executables are debugged, compiled and 
expert programmer is able to execute. 

Provide documentation of efforts. 

□ 

□ 

□ 

7. Piece parts and components in a pre- 
production form exist. Provide 
documentation of efforts. 

□ 

□ 

□ 

8. Production and integration planning have 
begun. Document planning efforts. 

□ 

□ 

□ 

9. Performance metrics have been established. 
Provide performance metrics. 

□ 

□ 

□ 

10. Cross technology issues (if any) have been 
fully identified. Document issues. 

□ 

□ 

□ 

11. Design techniques/codes have been defined 
to the point where medium level problems 
may be accommodated. Document level of 
fidelity and ownership of codes. 

□ 

□ 

□ 

12. Begin discussions/negotiations of 

Technology Transition Agreement to include 
data in items 1 through 5, 8, and 9. 

□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection 
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TECHNOLOGY READINESS LEVEL 5 

HARDWARE MATURITY CHECKLIST 

DoD TRL 5 
Definition 

TRL 5 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

Component 

and/or 

Breadboard 

Validation in 

Relevant 

Environment. 

Fidelity of 

breadboard 

technology 

increases 

significantly. The 

basic 

technological 
components are 
integrated with 
reasonably 
realistic 
supporting 
elements so that 
the technology can 
be tested in 
simulated 
environment. 
Examples include 
"high fidelity" 
laboratory 
integration of 
components. 

1. High fidelity lab integration of the hardware 
technology “system” completed and ready 
for testing in realistic simulated 
environments. Provide summary reports of 
integration efforts including results. Define 
relevant environment used in testing. 

□ 

□ 

□ 

Deputy for 

DV, Deputy 
for SE, and 
End User 
Deputy(ies) 
in 

consultation 
with Deputy 
for MP and 
Deputy 
Director for 
Technology 
and 

Engineering; 

ATC 

Secretariate 

informed. 

Certify that 
technology 
has met all 
applicable 

TRL 5 
Hardware 
Maturity 
Criteria and 
has thus 
achieved TRL 

5 status. 

2. Preliminary hardware technology 
“’’engineering report completed that 
addresses: 

a. Performance (including how measured 
performance translates to expected 
performance of final product) 

b. Integration 

c. Test and Evaluation 

d. Mechanical and Electrical Interfaces 

Provide preliminary hardware technology 

“system” engineering report. 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

3. Detailed design drawings have been 
completed. Three view drawings and 
wiring diagrams have been submitted. 

□ 

□ 

□ 

4. Pre-production hardware available. 

a. Prototypes have been created. 

b. Production processes have been reviewed 
with MDA/MP. Update ROM integration 
cost estimate and provide first order 
schedule for integration with end user(s). 

□ 

□ 

□ 

□ 

□ 

□ 

5. Form, fit, and function for application has 
begun to be addressed in conjunction with 
end user development staff. Provide details 
of efforts to date. 

□ 

□ 

□ 

6. Cross technology effects (if any) identified 
and established through analysis. Provide 
documentation of effects. 

□ 

□ 

□ 

7. Design techniques/codes have been defined 
to the point where largest problems defined. 
Provide details on how this technology will 
solve largest problems. 

□ 

□ 

□ 

8. Scaling studies have continued to next higher 
assembly from previous assessment. 

Describe scaling to new functional 
capability and regions of operational area. 

□ 

□ 

□ 

9. Technology Transition Agreement has been 
updated to reflect data in items 1 through 3, 

5, and 8. TTA has been coordinated and 
approved by ATC or end user Deputy(ies) 
and Deputy for DV following approval at 
CCB. 

□ 

□ 

□ 
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TECHNOLOGY READINESS LEVEL 5 

HARDWARE MATURITY CHECKLIST 

DoD TRL 5 
Definition 

TRL 5 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 


1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. _ 


TECHNOLOGY READINESS LEVEL 6 

HARDWARE MATURITY CHECKLIST 

DoD TRL 6 
Definition 

TRL 6 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

System/Subsystem 
Model or 

Prototype 
Demonstration in a 
Relevant 
Environment. 
Representative 
model or prototype 
system, which is 
well beyond the 
breadboard tested 
for level 5, is tested 
in a relevant 
environment. 
Represents a major 
step up in a 
technology’s 
demonstrated 
readiness. 

Examples include 
testing a prototype 
in a high-fidelity 
laboratory 
environment or in 
simulated 
operational 
environment. 

1. Materials, process, design, and integration 
methods have been employed. Provide 
documentation of process, design, and 
integration methodology compliance with 
MDA Quality Assurance Plan. 

□ 

□ 

□ 

Deputy for 

DV, Deputy 
for SE, and 
End User 
Deputy(ies) 
through 
MDA/DB 
and in 
consultation 
with Deputy 
for MP, 
Deputy 
Director for 
Technology 
and 

Engineering; 

ATC 

Secretariate 

informed. 

Certifies that 
technology 
has met all 
applicable 

TRL 6 
Hardware 
Maturity 
Criteria and 
has thus 
achieved TRL 

6 status. 

2. Scaling issues that remain are identified 
and supporting analysis is complete. 

Provide description of issues and 
resolution. 

□ 

□ 

□ 

3. Production demonstrations are complete. 
Production issues have been identified and 
major ones have been resolved. Provide 
documentation of data, issues and 
resolutions. 

□ 

□ 

□ 

4. Some associated “Beta” version software is 
available. 

□ 

□ 

□ 

5. Most pre-production hardware is available. 
Provide documentation of identified 
shortfalls to end user(s) and/or testing 
organization. 

□ 

□ 

□ 

6. Draft production planning has been 
reviewed by end user and developer. 

Update ROM integration cost estimate 
and update integration schedule with end 
user(s), MDA/SE, MDA/PI and MDAJMP. 

□ 

□ 

□ 

7. Draft design drawings are nearly complete. 

□ 

□ 

□ 

8. Integration demonstrations have been 
completed, including cross technology 
issue measurement and performance 
characteristic validations. Verification 
report compiled and reviewed by system 
engineer and testing organization. 

□ 

□ 

□ 

9. Have begun to establish an interface 
control process. Provide process 
documentation to system engineer for 
review. 

□ 

□ 

□ 
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TECHNOLOGY READINESS LEVEL 6 

HARDWARE MATURITY CHECKLIST 

DoD TRL 6 
Definition 

TRL 6 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 


10. Collection of actual maintainability, 
reliability, and supportability data has 
been started. Provide RAM data to 
system engineer. 

□ 

□ 

□ 


11. Representative model or prototype is 
successfully tested in a high- fidelity 
laboratory or simulated operational 
environment. Provide performance 
estimate and verification of capability 
enhancement with data collected. 

□ 

□ 

□ 

12. Hardware technology “system” 

specification complete. Submit hardware 
technology “ system ” specification for 
approval. 

□ 

□ 

□ 

13. Technology Transition Agreement has 
been updated to reflect data in items 1 
through 4, 7 through 9, 11 and 12. TTA 
has been coordinated and approved by 
ATC or end user Deputy(ies) and Deputy 
for DV following approval at CCB 

□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 


TECHNOLOGY READINESS LEVEL 7 

HARDWARE MATURITY CHECKLIST 

DoD TRL 7 
Definition 

TRL 7 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

System Prototype 
Demonstration in 
an Operational 
Environment. 

Prototype near or at 
planned operational 
system. Represents 
a major step up from 
level 6, requiring the 
demonstration of an 
actual system 
prototype in an 
operational 
environment. 
Examples include 
testing the prototype 
in a test bed aircraft. 

1. Materials, processes, methods, and design 
techniques have been identified and are 
moderately developed and verified. 

□ 

□ 

□ 

Cognizant 
Development 
Deputy in 
conjunction 
with SE. 
Certifies that 
technology 
has met all 
applicable 

TRL 7 
Hardware 
Maturity 
Criteria and 
has thus 
achieved TRL 

7 status. 

2. Scaling is complete. 

□ 

□ 

□ 

3. Production planning is complete. 

□ 

□ 

□ 

4. Pre-production hardware and software is 
available in limited quantities. 

□ 

□ 

□ 

5. Draft design drawings are complete. 

□ 

□ 

□ 

6. Maintainability, reliability, and 

supportability data growth is above 60% 
of total needed data. 

□ 

□ 

□ 

7. Hardware technology “system” prototype 
successfully tested in a field environment. 

□ 

□ 

□ 
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TECHNOLOGY READINESS LEVEL 7 
HARDWARE MATURITY CHECKLIST 


DoD TRL 7 
Definition 

TRL 7 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

1. For each criterion that HAS been met provide relevant background information for verification as 


noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 


TECHNOLOGY READINESS LEVEL 8 

HARDWARE MATURITY CHECKLIST 

DoD TRL 8 
Definition 

TRL 8 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

Actual System 
Completed and 
Qualified Through 
Test and 
Demonstration. 
Technology has 
been proven to work 
in its final fonn and 
under expected 
conditions. In 
almost all cases, this 
level represents the 
end of true system 
development. 
Examples include 
developmental test 
and evaluation of 
the system in its 
intended weapon 
system to detennine 
if it meets design 
specifications. 

1. Interface control process has been 

completed and final architecture diagrams 
have been submitted. 

□ 

□ 

□ 

Cognizant 
Development 
Deputy in 
conjunction 
with SE. 
Certifies that 
technology 
has met all 
applicable 

TRL 8 
Hardware 
Maturity 
Criteria and 
has thus 
achieved TRL 

8 status. 

2. Maintainability, reliability, and 

supportability data collection has been 
completed. 

□ 

□ 

□ 

3. Hardware technology successfully 
completes developmental test and 
evaluation. 

□ 

□ 

□ 

4. Hardware technology has been proven to 
work in its final form and under expected 
conditions. 

□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 


TECHNOLOGY READINESS LEVEL 9 

HARDWARE MATURITY CHECKLIST 

DoD TRL 9 
Definition 

TRL 9 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

Actual System 
Proven Through 
Successful Mission 
Operation. Actual 
application of the 

1. Hardware technology successfully 

completes operational test and evaluation. 

□ 

□ 

□ 

Cognizant 
Development 
Deputy in 
conjunction 
with SE. 

2. Training Plan has been implemented. 

□ 

□ 

□ 

3. Supportability Plan has been implemented 

□ 

□ 

□ 


60 






TECHNOLOGY READINESS LEVEL 9 

HARDWARE MATURITY CHECKLIST 

DoD TRL 9 
Definition 

TRL 9 Hardware Maturity Criteria 

Met 1 

Not 

Met 2 

N/A 3 

Certification 

Authority 

technology in its 
final form and 
under mission 
conditions, such as 
those encountered 
in operational test 
and evaluation. 
Examples include 
using the system 
under operational 
mission conditions. 

4. Program Protection Plan has been 
implemented 

□ 

□ 

□ 

Certifies that 
technology 
has met all 
applicable 

TRL 9 
Hardware 
Maturity 
Criteria and 
has thus 
achieved TRL 

9 status.. 

5. Safety/Adverse effects issues have been 
identified and mitigated 

□ 

□ 

□ 

6. Operational Concept has been 

implemented successfully 

□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 

1. For each criterion that HAS been met provide relevant background information for verification as 
noted. 

2. For each criterion that HAS NOT been met provide the status and an estimate when the criteria will 
be met. 

3. For each criterion marked N/A provide supporting documentation for this selection. 


Table 4. Missile Defense Agency Hardware Technology Readiness Level Checklist 
[From: Missile Defense Agency, 2007] 


A Technology Readiness Calculator was developed by William Nolte who works 
for the Air Force Research Laboratory. It is available at the DoD Acquisition 
Community Connection website; Version 2.2 is available at 
https://acc.dau.mil/CommunityBrowser.aspx?id=2581 l&lang=en-US. It is a Microsoft 
Excel™ spreadsheet application with a standard set of questions about hardware, 
software, manufacturing, engineering artifacts, and the acquisition program. It calculates 
and graphically displays a TRL achieved. This calculator is unique in that it looks at 
technology holistically within the context of the software and hardware technology itself, 
manufacturing, system development and programmatics. See Figure 22 for the 
calculator. In discussion with the Institute of Defense Analyses which supports the 
Missile Defense Agency, they are looking into creating a TRL calculator as well. 
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AFRL Transition Readiness Level Calculator, version 2.2 


Summary 


1 (§) Use Manufacturing 

1 0 No Manufacturing 

Hide 
□ Blank 

Rows 

1 (?) Use Programmatics 

1 0 No Programmatics 

% Complete is 
now set at: 

100% 


Green set point is: 100% Yellow set point is: 67% Change set points on Summary sheet. 


Hardware and Software Calculator 

Technology Readiness Level Achieved 

Technical: 


1 1 2 1 3 | 4 | — 

6 i 7 

8 9 


1 1 


|0 Only Hardware 
[o^ Software 


(•) Hardware & Software 


Program Name: 


Date TRL Computed: 


Program Manager: 



TOP LEVEL VIEW -- Demonstration Environment (Start at top and pick the first correct answer) 



® 

Has an identical unit been successful an on operational mission (space or launch) in an identical configuration? 



0 

Has an identical unit been demonstrated on an operational mission, but in a different configuration/system architecture? 



U 

Has an identical unit been mission (flight) qualified but not operationally demonstrated (space or launch)? 

TRL 9 


0 

Has a prototype unit been demonstrated in the operational environment (space or launch)? 



0 

Has a prototype been demonstrated in a relevant environment, on the target or surrogate platform? 



0 

Has a breadboard unit been demonstrated in a relevant (typical; not necessarily stressing) environment? 



0 

Has a breadboard unit been demonstrated in a laboratory (controlled) environment? 



0 

Has analytical and experimental proof-of-concept been demonstrated? 



0 

Has a concept or application been formulated? 



0 

Have basic principles been observed and reported? 



0 

None of the above 



1 Source: James W. Bilbro, NASA, Marshall SFC, May 2001| 


Comments: | 

i 




H/SW 

Both 

Uues ■' : — 1 - " ■■ ‘ - 

Catgry | % Complete 

TRL 1 (Check all that apply or use slider for % complete) 

B 

t S 21 


a 

"Back of envelope" environment 

B 

t K 21 

R! 

M 

Physical laws and assumptions used in new technologies defined 

S 

T 11 E 

Rf 

h 

Have some concept in mind that may be realizable in software 

S 

T K. 2 

Rf 

0 

Know what software needs to do in general terms 

B 

T Lft 

Rf 

0 

Paper studies confirm basic principles 

S 


Rf 

0 

Mathematical formulations of concepts that might be realizable in software 

S 

T E E 

Rf 

0 

Have an idea that captures the basic principles of a possible algorithm 

B 

p £ 2 

Rf 

0 

Initial scientific observations reported in journals/conference proceedings/technical reports 

B 

T £2 

Rf 

R 

Basic scientific principles observed 

B 

p It 2 

rre 

0 

Know who cares about technology, e.g., sponsor, money source 

B 

T £ 12 

Rf 

Hi 

Research hypothesis formulated 

B 

p It 2 

Rf 

a 

Know who will perform research and where it will be done 


It 2 

Rf 

0 



K 2 

Rf 

0 



« >1 

UK 

0 



It 2 

100 

0 



i S 

100 

0 



Comments: 
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H/SW 

Both 

Ques 

Catgry 


% Complete 

TRL 2 (Check all that apply or use slider for % complete) 

B 

P 

K 

3 100 


Customer identified 

B 

T 


ee 

w 

Potential system or component application(s) have been identified 

B 

T 

K 

3100 

R 

Paper studies show that application is feasible 

B 

P 

1 

an ioo 

R 

Know what program the technology will support 

B 

T 

£ 

aiioo 

R 

0 

0 

An apparent theoretical or empirical design solution identified 

H 

T 

£ 

>J[! 

Basic elements of technology have been identified 

B 

T 

[ 

I 

Desktop environment 

H 

T 

£ 

^ii • 

0 

Components of technology have been partially characterized 

H 

T 

£ 


0 

Performance predictions made for each element 

B 

P 

K — 


0 

Customer expresses interest in application 

S 

T 

£ 


0 ., 

Some coding to confirm basic principles 

B 

T 

K 

ill! 

0 

Initial analysis shows what major functions need to be done 

H 

T 

£ 

bi' 

R 

0 

Modeling & Simulation only used to verify physical principles 

B 

P 

£ 

I! 

System architecture defined in terms of major functions to be performed 

S 

T 



0 

Experiments performed with synthetic data 

B 

P 

£ 

iiE! 

0 

Requirement tracking system defined to manage requirements creep 

B 

T 

< 

jl! 

0 

Riqorous analytical studies confirm basic principles 

B 

P 

if 

E! 

0 

Analytical studies reported in scientific journals/conference proceedinqs/technical reports 

B 

T 

< 

>|! 

0 

Individual parts of the technoloqy work (No real attempt at integration) 

S 

T 

if 


0 

Know what hardware software will be hosted on 

B 

T 

< 

! 

0 

Know what output devices are available 

B 

P 

< 

lE! 

R 

Investment Strategy Sheet 

B 

P 

< 

>E' 

0 

Know capabilities and limitations of researchers and research facilities 

B 

T 

£ 

H ! 

0 

Know what experiments you need to do (research approach) 

B 

P 

£ _ ME 

0 

□ 

0 

, 0 ! 

0 

0 

0 

0 

0 

Qualitative idea of risk areas (cost, schedule, performance) 

B 

P 

It — 

2E 

Have rough idea of how to market technology (Who's interested, how will they find out about it?) 



£ 

>jlg 




< 

SiB 




< 





< 

SB 









_ 





if 

SB 
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H/SW 

Ques 

I" 



Both 

Catgry 

| % Complete 

TRL 3 (Check all that apply or use slider for % complete) 

B 

T 

X ft 


0 

0 

Academic environment 

H 

T 

It It 



0 

Predictions of elements of technology capability validated by Analytical Studies 

S 

T 

J*ft 



0 

Analytical studies verify predictions, produce algorithms 

H 

T 

|t ft 



0 

Science known to extent that mathematical and/or computer models and simulations are possible 

H 

P 




0 

Preliminary system performance characteristics and measures have been identified and estimated 

S 

T 

i 



0 

Outline of software algorithms available 

H 

T 

It ft 

D* 


0 

Predictions of elements of technology capability validated by Modeling and Simulation 

S 

T 

It ft 



a 

Preliminary coding verifies that software can satisfy an operational need 

H 

M 

K > 



0 

No system components, just basic laboratory research eguipment to verify physical principles 

B 

T 

l< > 



0 

Laboratory experiments verify feasibility of application 

H 

T 

It ft 

r! 


0 

Predictions of elements of technology capability validated by Laboratory Experiments 

B 

P 

l< l> 



0 

Customer representative identified to work with development team 

B 

P 

It ft 

e$f 


a 

Customer participates in reguirements generation 

B 

T 

|t ft 



a 

Cross technology effects (if any) have begun to be identified 

H 

M 

L L 



b 

Design technigues have been identified/developed 

B 

T 




0 , 

Paper studies indicate that system components ought to work together 

B 

P 




IB 

Customer identifies transition window(s) of opportunity 

B 

T 

|t ft 



0 

Metrics established 

B 

P 

T ft 

iff 


0 

Scaling studies have been started 

S 

T 

1 ft 

R 


R 

Experiments carried out with small representative data sets 

S 

T 

It ft 

rf 


0 

Algorithms run on surrogate processor in a laboratory environment 

H 

M 

It ft 

R 


0 

Current manufacturability concepts assessed 

S 

T 

X i 

R 


0 

Know what software is presently available that does similar task (100% = Inventory completed) 

S 

T 

x ft 

R 


0 

Existing software examined for possible reuse 

H 

M 

t > 



0 

Producibility needs for key breadboard components identified 

S 

T 

f 

R 


0 

Know limitations of presently available software (Analysis of current software completed) 

B 

T 

t ft 

R 


0 

Scientific feasibility fully demonstrated 

B 

T 

K > 



0 

Analysis of present state of the art shows that technology fills a need 

B 

P 

t ft 



0 

Risk areas identified in general terms 

B 

P 

O 



0 

Risk mitigation strategies identified 

B 

P 

t ft 



0 

Rudimentary best value analysis performed, not including cost factors 


£ ft 



0 



t ft 

R 


0 






0 



K. ft 



0 



K > 


1 

ET 



It ft 


0 

01 



K 1 


a 

0 
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H/SW 

Ques 


Both 

Catgry 

% Complete 


TRL 4 (Check all that apply or use slider for % complete) 

B 

T 

< 

HI 


□ 

Cross technology issues (if any) have been fully identified 

H 

M 

£IT 

00 

□ 

Ad hoc and available laboratory components are surrogates for system components 

H 

T 

£ 

00 

0 

Individual components tested in laboratory/by supplier (contractor's component acceptance testinq) 

H 

M 

K 

Jr 

00 

0 

Piece parts and components in a pre-production form exist 

H 

T 

« 

ill 

00 

0 

M&S used to simulate some components and interfaces between components 

S 

T 

£ M 

00 

0 

Formal system architecture development beqins 

B 

P 

1 

□L 

00 

0 

Customer publishes requirements document 

B 

T 

< M 

00 

0 

Overall system requirements for end user's application are known 

B 

P 

C >1 

00 

0 

System performance metrics have been established 

S 

T 

< i>B 

00 

0 

Analysis provides detailed knowledqe of specific functions software needs to perform 

B 

P 

< a 

00 

0 

Laboratory requirements derived from system requirements are established 

H 

M 

c >iE 

00 

0 

Available components assembled into system breadboard 

H 

T 

i>| 

00 

0 

Laboratory experiments with available components show that they work together (lab kludge) 

S 

T 

x iS 

00 

0 

Requirements for each function established 

S 

T 

K 

> 

00 

□ 

Algorithms converted to pseudocode 

S 

T 

* 


00 

□ 

Analysis of data requirements and formats completed 

S 

T 

K 

i 

00 

EL 

Stand-alone modules follow preliminary system architecture plan 

H 

T 

< 


00 

0 

Hardware in the loop/computer in the loop tools to establish component compatibility 

S 

M 

X 

i 

00 

0 

Designs verified throuqh formal inspection process 

B 

P 

X 

A 

00 

FI 

S&T exit criteria established 

B 

T 

X 

- 

00 

0 

Technology demonstrates basic functionality in simplified environment 

S 

P 

i 

- 

00 

0 

Able to estimate software proqram size in lines of code and/or function points 

H 

M 

K >|E 

00 

0 

Scalable technology prototypes have been produced 

B 

P 

X 

00 

0 

Draft conceptual desiqns have been documented 

H 

M 

k lijE 

00 

0 

Design techniques identified/defined to where small applications may be analyzed/simulated 

B 

T 

x iS 

00 

0 

Controlled laboratory environment 

B 

P 

A if 

00 

0 

Initial cost drivers identified 

S 

T 

x i>|i 

00 

0 

Experiments with full scale problems and representative data sets 

B 

M 

K 

A 

00 

0 

Integration studies have been started 

B 

P 

It 


00 

0 

CAIV tarqets set 

S 

T 

£ 

£ 

00 

EF 

Individual functions or modules demonstrated in a laboratory environment 

H 

M 

* 

- 

00 

s.: 

Key manufacturing processes identified 

B 

P 

K 


00 

SL 

Scalinq documents and diaqrams of technology have been completed 

S 

T 

K 


00 

m 

Some ad hoc integration of functions or modules demonstrates that they will work toqether 

H 

M 

K 

. 

00 

af" 

Key manufacturing processes assessed in laboratory 

B 

P 

< 

. 

00 

EL 

Draft Systems Enqineerinq Master Plan (SEMP) 

B 

T 

£ It 

00 

a 

Low fidelity technology "system” integration and engineering completed in a lab environment 

H 

M 

£ It 

00 

2L 

Mitigation strategies identified to address manufacturability / producibility shortfalls 

B 

P 

£ A 

00 

0 

Customer commits to transition throuqh ATD commissioning and/or MOU 

B 

T 

It A 

00 

0 

Functional work breakdown structure developed 

B 

P 

« > 

00 

0 

Integrated Product Team (IPT) formally established with charter 

B 

P 

A A 

00 

ET 

Customer representative is member of IPT 

B 

P 

£ 

AL 

00 

0 

Formal risk management program initiated 

B 

P 

L _AL 

00 

0 

Preliminary Failure Mode and Effects Analysis (FMEA) or Risk Waterfall analysis performed 

B 

P 

£ 


00 

0 

Technology availability dates established 



twrl 


00 

0 






00 

0“ 




£ M3 

00 

FT 




1 Ip 

00 

0 




E _SI 

00 

FT 




£ _ 

AL 

00 

0 




1 1 

X 

00 

0 
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H/SW 

Ques 


Both 

Catgry 

% Complete 

TRL 5 (Check all that apply or use sliders) 

B 

T 

* Van 

0 

Cross technology effects (if any) identified and established through analysis 

H 

M 


SB 

! Ld 

Pre-production hardware available 

B 

T 

< 

as 

0 

System interface requirements known 

B 

P 

£ ag 

! □ 

System requirements flow down through work breakdown structure (systems engineering begins) 

S 

T 

£ In 

3] Ld 

System software architecture established 

H 

M 

2_JH 

0 

Targets for improved yield established 

S 

T 

± 


0 

External interfaces described as to source, format, structure, content, and method of support 

S 

T 


>jB 

HT 

Analysis of internal interface requirements completed 

H 

M 



0 

Trade studies and lab experiments define key manufacturing processes 

B 

T 

_ 

K 

> 

H 

Interfaces between components/subsystems are realistic (Breadboard with realistic interfaces) 

H 

M 

< >b 

0 

Significant engineering and design changes 

S 

T 

c 


0 

Coding of individual functions/modules completed 

H 

M 

£ 

1 (j 

0 

Prototypes have been created 

H 

M 



0 

Tooling and machines demonstrated in lab 

B 

T 

4 _ tn 

0 

High fidelity lab integration of system completed, ready for test in realistic/simulated environments 

H 

M 

£ SI 

0 

Design techniques have been defined to the point where largest problems defined 

H 

P 

£ 


0 

Form, fit, and function for application addressed in conjunction with end user development staff 

H 

T 

£ 

J 

0 

Fidelity of system mock-up improves from breadboard to brassboard 

B 

M 

« 

J 

0 

Quality and reliability considered, but target levels not yet established 

H 

M 

< 

£ 

MS' 

Some special purpose components combined with available laboratory components 

H 

P 

< 

> 

0 

Three view drawings and wiring diagrams have been submitted 

B 

T 


! [7] 

Laboratory environment modified to approximate operational environment 

H 

M 

< 

>1 

I J 

Initial assesment of assembly needs performed 

H 

P 

< 

>| 

0 

Detailed design drawings have been completed 

H 

M 

< 

3 

0 

Sigma levels needed to satisfy CAIV targets defined 

B 

P 

£ 

>i ioo a 

Draft SEMP addresses integration 

B 

P 

£ 

> 

0 

Draft SEMP addresses test and evaluation 

B 

P 

i 


0 

Draft SEMP addresses mechanical and electrical interfaces 

H 

M 


> 

0 

Production processes have been reviewed with Manufacturing and Producibility office(s) 

B 

P 

* 

> 

if 

Draft SEMP addresses performance; translate measured to expected final performance 

B 

P 

K 

) 

0 

Risk management plan documented 

S 

T 



! pi 

Functions integrated into modules 

B 

P 

< 

> 


Configuration management plan in place 

S 

T 

£ 

> 

0 

Individual functions tested to verify that they work 

S 

T 

K 

> 

i □ 

Individual modules and functions tested for bugs 

S 

T 

{ 

> 

, [vj 

Integration of modules/functions demonstrated in a laboratory environment 

S 

P 

< 

> 

! 0 

Formal inspection of all modules/components completed as part of configuration management 

B 

P 

£ m 

0 

Configuration management plan documented 

B 

P 

£ 

iB 

0 

Draft Test & Evaluation Master Plan (TEMP) 

S 

T 

£ 


0 

Algorithms run on processor with characteristics representative of target environment 

H 

P 

£ 

ifl 

0 

Preliminary hardware technology “system" engineering report (Draft SEMP) completed 

B 

P 

K 

> 

0 

Customer commits to transition via POM process 

B 

P 

< 

> 

0 

Draft Transition Plan with Business Case 

H 

P 

£ >||] 

0 

Failure Mode and Effects Analysis (FMEA) performed 

B 

P 

£ >| ] 

: 0 

Value analysis includes analysis of multiple technology and non-material alternatives 

B 

T 

£ 

SB 

0 

IPT develops requirements matrix with thresholds and objectives 

B 

T 

4_i| 

0 

Physical work breakdown structure available 

B 

P 

£ « 

0 

Value analysis includes life-cycle cost analysis 



t 

m 

j 0 




£ a 

0 




£ 


0 




£ 

J 

a 




£ 

Si 

a; 




£ 

1 

a: 




£ 

jl 

3 




£ 

jl_ 

I 3 




£ 


! 3T 
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H/SW 

Both 

Ques 

Catgry 


% Complet 

TRL 6 (Check all that apply or use sliders) 

B 

T 

It ME 

R 

Cross technology issue measurement and performance characteristic validations completed 

H 

M 

i W. 

0 

Quality and reliability levels established 

B 

M 


0 

Frequent design changes occur 

H 

P 


0 

Draft design drawinqs are nearly complete 

B 

T 

1C ME 

0 

Operating environment for eventual system known 

B 

P 

& M 

0 

Collection of actual maintainability, reliability, and supportability data has been started 

B 

P 

K Ml 

0 

Desiqn to cost qoals identified 

H 

M 

1C ME 

0 

Investment needs for process and tooling determined 

B 

T 

* a 

0 

M&S used to simulate system performance in an operational environment 

B 

P 

< X 

0 

Final Test & Evaluation Master Plan (TEMP) 

H 

T 

i 1 

0 

Factory acceptance testing of laboratory system in laboratory setting 

B 

T 

< > 

0 

Representative model / prototype tested in high-fidelity lab / simulated operational environment 

B 

T 

I 1 

0 

Realistic environment outside the lab, but not the eventual operating environment 

B 

P 

* > 

0 

Final Systems Engineering Master Plan (SEMP) 

s 

T 

s I 

0 

Inventory of external interfaces completed 

B 

P 


nr 

Technology Transition Agreement has been updated 

B 

P 

* > 

0 

Scaling issues that remain are identified and supporting analysis is complete 

s 

T 

t i 

a 

Analysis of timing constraints completed 

s 

T 

< ? 

GT 

Analysis of database structures and interfaces completed 

B 

P 

i > 

ET 

Have begun to establish an interface control process 

H 

P 

ri 

0 

Draft production planning has been reviewed by end user and developer 

H 

M 

K l> 

0 

Critical manufacturing processes prototyped 

H 

M 

K > 

0 

Most pre-production hardware is available 

B 

P 


0 

Technology Transition Agreement has been coordinated and approved by end user 

s 

T 

ri 

0 

Prototype implementation includes functionality to handle large scale realistic problems 

s 

T 

? ? 

0 

Algorithms parially integrated with existing hardware / software systems 

H 

M 

* > 

nr 

Materials, process, design, and integration methods have been employed 

s 

T 

< > 

0 

Individual modules tested to verify that the module components (functions) work together 

B 

P 

i 1 

0 

Technology "system” specification complete 

H 

M 

irn 

0 

Components are functionally compatible with operational system 

s 

T 

i|i 

0 

Representative software system or prototype demonstrated in a laboratory environment 

B 

T 

< > 

0 

Laboratory system is high-fidelity functional prototype of operational system 

B 

P 

< > 

0 

Formal configuration management program defined to control change process 

B 

M 

« > 

0 

Integration demonstrations have been completed 

B 

P 

Q 

0 

Final Technical Report 

H 

M 

K > 

□ 

Production issues have been identified and major ones have been resolved 

s 

T 

< > 

Ld 

Limited software documentation available 

s 

P 

< > 

0 

Verification, Validation and Accreditation (VV&A) initiated 

H 

M 

* > 

0 

Process and tooling are mature 

H 

M 

1 | 

0 

Production demonstrations are complete 

s 

P 


0 

"Alpha" version software has been released 

B 

T 


0 

Engineering feasibility fully demonstrated 

B 

P 

is. i 

0 

Final Transition Plan with Business Case 

B 

P 

t i 

0 

Acquisition program milestones established 

B 

P 

ri 

E 

Value analysis includes business case 

B 

P 


H 

Technical alternatives include "do nothing case" 

B 

P 


0 

Formal requirements document available 



«. M 

0 





0 





B 





0 





0 




< 3 

0 




S 

... 0 





0 




S ME 

nr 
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H/SW 

Ques 


Both 

Catgry 

% Complete 

TRL 7 (Check all that apply or use sliders) 

H 

M 

< 

>E 

kl 

Materials, processes, methods, and design techniques have been identified 

H 

M 

K 

i 

E 

ST 

Materials and manufacturing process and procedures initially demonstrated 

H 

T 

* 


a 

M&S used to simulate some unavailable elements of system, but these instances are rare 

H 

M 

£ 


R 

Prototype system built on "soft" tooling 

B 

T 

£ 

i 


0 

Each system/software interface tested individually under stressed and anomolous conditions 

S 

T 

£ 


Iff 

R 

Algorithms run on processor(s) in operating environment 

S 

P 

£ a 

R 

VV&A in process with the verification step that software specifications are met completed 

H 

M 

£ 1 

R 

Process tooling and inspection / test equipment demonstrated in production environment 

H 

M 

£ 


100 

0 

Machines and tooling proven 

H 

M 

< 

l 

EET 

ibl 

Desiqn chanqes decrease siqnificantly 

B 

T 

£ 

>i 

% 

Operational environment, but not the eventual platform, e.q., test-bed aircraft 

B 

M 

£ 11 P' 

H 

Maintainability, reliability, and supportability data is above 60% of total needed data 

H 

P 

< >| n 1 

R 

Draft desiqn drawings are complete. 

H 

M 


R 

Materials, processes, methods, and desiqn techniques are moderately developed and verified 

B 

P 

< 

> 

100 

R 

Scaling is complete. 

H 

M 

& 

1 


a 

Pre-production hardware is available; quantities may be limited 

H 

T 

< 

a 

K 

a 

Components are representative of production components 

H 

P 

< 

IfB 

if 

Desiqn to cost qoals validated 

H 

M 



n 

Initial sigma levels established 

H 

M 

£ 

5 

EE 

R 

Manufacturing processes qenerally well understood 

S 

M 

£ m. 

H 

Most software bugs removed 

H 

M 

£ 

> 

100 

R 

Production planning is complete. 

B 

T 

£ 


100 

H 

Most functionality available for demonstration in simulated operational environment 

B 

T 

< 

1 

100 

H 

Operational/flight testing of laboratory system in representational environment 

H 

M 

< 

1 

EE 

R 

Prototype improves to pre-production quality 

S 

P 

£ 

-m 

R 

"Beta" version software has been released 

B 

T 

£ 

r 

100 

H 

Fully integrated prototype demonstrated in actual or simulated operational environment 

B 

T 

£ 

u 

100 

R 

System prototype successfully tested in a field environment. 

H 

M 

£ 

2 

m 

R 

Ready for Low Rate Initial Production (LRIP) 



£ 


100 

R 




1 


IS 

R 




£ 

I 

100 

R 




£ 

i 


R 




£ 

i 


H 






R 





2 


H 
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H/SW 

Ques 


Both 

Catgry 

% Complete 

TRL 8 (Check all that apply or use sliders) 

B 

T 

K 

> 


a 

Components are form, fit, and function compatible with operational system 

H 

M 

K 

y 


0 

Cost estimates <125% cost goals (e.g., design to cost goals met for LRIP) 

B 

T 

< 

> 

100 

0 ' 

System is form, fit, and function design for intended application and weapon system platform 

B 

T 

K 

b 

100 

0 

Form, fit, and function demonstrated in eventual platform/weapon system 

H 

M 

£ 

b 


0 

Machines and tooling demonstrated in production environment 

B 

T 


a 


0 

Interface control process has been completed 

S 

P 

< 

b 

100 

3 

Most software user documentation completed and under configuration control 

B 

P 

< 



0 

Most traininq documentation completed and under configuration control 

B 

P 

rt 

£ 


0 

Most maintenance documentation completed and under configuration control 

B 

T 

)L 

l£l 


0 

Final architecture diaqrams have been submitted 

H 

M 

K 

y 

100 

0 

Manufacturing processes demonstrated by pilot line, LRIP, or similar item production 

H 

M 

£. 

b 


0 

Manufacturing processes demonstrate acceptable yield and producibility levels 

S 

T 

£ 

b 


0 

Software thoroughly debuqqed 

B 

T 


100 

0 

All functionality demonstrated in simulated operational environmenet 

H 

M 


100 

0 

Manufacturing process controlled to 4-sigma or appropriate quality level 

H 

M 

£ 

n 

100 

0 

All materials are in production and readily available 

B 

T 

&M 

100 

0 

System qualified through test and evaluation on actual platform (DT&E completed) 

B 

M 

it 


100 

0 

Maintainability, reliability, and supportability data collection has been completed 

S 

P 

t b 

100 

0 

VV&A validation step completed, software works in real world 

B 

T 


i 

100 

0 

DT&E completed, system meets specifications 

S 

P 

I 

b 

100 

0 

VV&A accreditation step completed, software authorized for use in intended weapon system 

H 

M 


100 

0 

Ready for Full Rate Production 



x b 

100 

0 





b 

100 

0 





± 

100 

0 





b 

100 

0 




< 

b 

100 

0 




tL 

Jj 

100 

0 
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H/SW 

Ques 

_ 

Both 

Catgry 

% Complete 

TRL 9 (Check all that apply or use sliders) 

B 

T 

4 

b 

10 

0 

E 

Operational Concept has been implemented successfully 

H 

M 

4 

i 

10 

0 

0 

Cost estimates <110% cost goals or meet cost goals (e.g., design to cost goals met) 

H 

M 

r 

zf 

10 

0 

0 

Affordability issues built into initial production and evolutionary acquisition milestones 

H 

M 

< 

> 


g 

0 

Design stable, few or no design changes 

B 

T 




0 

0 

System has been installed and deployed in intended weapon system platform 

B 

P 

s r 

i 


0 

0:j 

Safety/Adverse effects issues have been identified and mitigated. 

B 

T 

ST 

i 


0 

0: 

Actual system fully demonstrated 

B 

P 


b 


0 

a 

Training Plan has been implemented. 

B 

P 

4 

4 



0 

Supportability Plan has been implemented. 

B 

P 

ST 




0 

Program Protection Plan has been implemented. 

B 

T 

< 

b 



0 

Actual mission system "flight proven" through successful mission operations (OT&E completed) 

H 

M 

£ 

i| 



0 

All manufacturing processes controlled to 6-sigma or appropriate quality level 

H 

M 


AH 



0 

Stable production 

B 

P 

LJ 


0 

0 

All documentation completed 
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ft 



EL 
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0 



Comments: [ 


Figure 21. AFRL Nolte TRL Calculator v2.2 [From: Nolte, 2004] 


H - Hardware 
S - Software 


M - Manufacturing 
B - Both H and S 


T - Technical 
P - Programmatics 
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The GAO assesses DoD programs on a regular basis. Their findings consistently 
find programs using immature technologies sometimes through MS C. In one report of 
many, GAO-05-301 Defense Acquisitions: Assessments of Selected Major Weapons 
Programs, March 31, 2005 (GAO, 2005) had the following summary of a review of 54 
DoD programs: 

• Only 15% of programs began System Design and Development (SDD) 
with mature technology (TRL 7) 

• Programs that started with mature technologies averaged 9% cost growth 
and a 7 month schedule delay; Programs that did not have mature 
technologies averaged 41% cost growth and a 13 month schedule delay 

• At critical design review, 42% of programs demonstrated design stability 
(90% drawings releasable); Design stability not achievable with immature 
technologies 

• Programs with stable designs at CDR averaged 6% cost growth; Programs 
without stable designs at CDR averaged 46% cost growth and a 29 month 
schedule delay 

Given the track record of DoD programs selecting immature technologies, 
Congress passed legislation in 2006: 801 - the Title VIII—Acquisition Policy, 
Management, and related matters House Conference Report 109-360 (United States 
House of Representatives, 2006) that requires the MDA for MDAP/MAIS programs to 
certify, among other things, that all technologies have been demonstrated in a relevant 
environment (TRL 6) for all technologies prior to MS B. 

2. Non DoD Industry 

There is very little written about formal technology readiness assessments by 
industry. GAO/NSAID report 99-162 (GAO, 1999) interviewed commercial industry and 
found that in general Industry waits until a technology is equivalent to a TRL 8 before 
integration into a product. The report contends that: 

.. .leading commercial firms’ practices have produced results that resemble 
those sought by DOD: more technically advanced, higher quality products, 
developed in significantly less time, and less expensively than their 
predecessors...The commercial firms “managing the development of 
advanced technology differently—and separately—from the development of 
a product has been key to these results. The firms insist that advanced 
technology reach a high level of maturity, the point at which the 
knowledge about that technology is essentially complete, before allowing 
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it into a product development. By separating the two, the firms lessen the 

product manager’s burden and place that person in a better position to 

succeed in delivering the product. 

In working with the Venture Capital community during 2003, I found that the 
Venture Capitalists (VCs) and other technology incubators use an informal assessment 
process much like the DoD’s independent review process by the S&T Executives except 
the assessment in done in the context of business and market due diligence analogous to 
DoD’s operational environment. 

3. Academia 

Sauser et al. from the Stevens Institute of Technology has proposed two 
additional readiness levels in his paper ‘From TRL to SRL: The Concept of Systems 
Readiness Levels (Sauser et al., 2006).’ Sauser et al. contends that the use of the 
NASA/DoD TRLs doesn’t take in account the technology within a system, the 
interactions between the combination of technologies within a system and the 
interoperability between systems. 

Sauser et al. defines System Readiness Levels (SRL) to be defined by the current 
state of development of a system per DoDs acquisition phases of system development 
(e.g., SDD). The SRL is a function of the individual Technology Readiness Levels 
(TRL) in a system and their subsequent integration points with other technologies, called 
an Integration Readiness Level (IRL). See Figure 23 for a full list of SRL levels and 
definitions. 

Sauser et al. defines the IRL as systematic measurement of the compatible 
interactions for various technologies and the consistent comparison of the maturity 
between integration points. It’s a measure of the maturity of combining and coordinating 
of separate components into a seamless unit. See Figure 24 for a list of IRL levels and 
definitions. 

Sauser et al. has a concept as seen in Figure 25 of how SRL is a function of 
component TRLs and the IRLs between them. Sauser et al. is continuing to work on 
these concepts. 
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SRL 

Name 

Definition 

5 

Operations & Support 

Execute a support program that meets 
operational support performance 
requirements and sustains the system in 
the most cost-effective manor over its 
total life cycle. 

4 

Production & Development 

Achieve operational capability that 
satisfies mission needs. 

3 

System Development & 
Demonstration 

Develop a system or increment of 
capability; reduce integration and 
manufacturing risk; ensure operational 
supportability; reduce logistics footprint; 
implement human systems integration; 
design for producibility, ensure 
affordability and protection of critical 
program information: and demonstrate 
system integration, interoperability, 
safety, and utility. 

2 

Technology Development 

Reduce technology risks and determine 
appropriate set of technologies to 
integrate into a full system. 

1 

Concept Refinement 

Refine initial concept. Develop 
system/technology development 
strategy 


Figure 22. System Readiness Levels [From: Sauser et al., 2006] 


IRL 

Definition [9] 

7 

The integration of technologies has been verified and validated with sufficient 
detail to be actionable. 

6 

The integrating technologies can accepr, translate, and structure information 
for its intended application 

5 

There is sufficient control between technologies necessary to establish, 
manage, and terminate the integration. 

4 

There is sufficient detail in the quality and assurance of the integration between 
technologies. 

3 

There is compatibility (i.e. common language) between technologies to orderly 
and efficiently integrate and interact 

2 

There is some level of specificity to characterize the interaction (i.e. ability to 
influence) between technologies through their interface. 

1 

An interface (i.e physical connection) between technologies has been identified 
with sufficient detail to allow characterization of the relationship. 


Figure 23. Integration Readiness Levels [From: Sauser et al., 2006] 
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Figure 24. SRL and TRL and IRL [From: Sauser et al., 2006] 
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III. RESEARCH 


A. RESEARCH APPROACH 

Definitions of SoS and FoS and a taxonomy for degrees of interoperability are 
derived from the literature review. Requirements and guidelines for conducting a SoS 
TRA will be developed. A short checklist will be developed for identifying where SoS 
technologies are located to facilitate analyzing the number of systems that are part of the 
SoS and the potential for unexpected results. The ‘system’ definitions, interoperability 
taxonomy, SoS TRA requirements and guidelines and SoS technology locator checklist 
will be used in the analysis of four possible SoS with respect to the thesis research 
questions. 

Research questions: 

1. What are the appropriate definitions of SoS in the context of conducting 
TRAs? [Section III B.] 

2. What are the appropriate definitions for interoperability and its use in 
defining the operational relevant environment for conducting SoS TRAs? 
[Section III C.] 

3. What is the approach for determining critical technology elements for 
SoS? [Section III D.] 

4. What are the fundamental requirements and guidelines for conducting a 
SoS TRA and how are these different from a system TRA? [Section III D.] 

5. What technology development and acquisition strategies should be 
employed for technology maturation for SoS given the challenges of 
synchronization of individual system acquisition schedules? [Identify 
challenges during FoS/SoS analysis] 

6. When is the ‘right’ time to hold SoS acquisition milestones given the 
synchronization issues with the individual systems that make up the SoS? 
[Identify challenges during FoS/SoS analysis] 

Research questions 5 and 6, respectively are facilitated by analysis of the 
technical and systemic challenges. This research will answer whether the SoS/FoS under 
analysis experienced any of these challenges and how they handled them. 
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Technical challenges: 

a) Capability requirements and functional analysis should occur prior to 
specific system requirements, system functional analysis, and system 
technology development; however, many SoS are assembled from legacy 
systems and network-centric functionality may be constrained 

b) Key Perfonnance Parameters (KPPs) or operational requirements for a 
capability are not easily allocated to individual systems and their 
subsystems 

c) Appropriate SoS relevant environment modeling and simulation and test 
and evaluation environments will typically be built post system design and 
development 

d) Identification of technology elements given the degree of interoperability 
or integration may not be obvious within a (re)composable context or 
environment 

e) SoS are typically enabled with software which is easily changed 
incrementally over time 

Systemic challenges: 

a) Critical technology developed by the individual programs are in alignment 
with their respective schedules not the SoS program schedule 

b) SoS technology selections and development prior to completion of 
capability engineering and then individual system(s) engineering drives up 
risk; SoS engineering needs to be at least through System Functional 
Review prior to a MS B decision 

c) It’s challenging to test the critical technologies in an integrated manner if 
the individual systems have not had the opportunity to all develop their 
systems enough to have representative systems for SoS testing (e.g., 
relevant environment for a integrated heterogeneous distributed system) 

d) The fielding of a SoS capability is typically time-phased over several 
years in capability spirals or increments with differing sets of systems and 
services 

B. ‘SYSTEM’ DEFINITIONS 

There clearly is no one set of consistent and agreed to terminology for system, 
FoS and SoS or the types thereof. If one is conducting a TRA it is imperative to have an 
understood definition in order to characterize the operational environment. A definition 
provides the basis for identifying the boundary where KPPs/operational requirement will 
be measured and therefore the boundary that encompasses the CTEs that support 
achieving the specified KPPs. 
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The definition of system that would support the definitions of SoS selected from 
the literature review is the following from Maier and Rechtin. 

System: 

...a collection of things or elements which, working together, produce a 
result not achievable by the things alone. 

This definition was selected based on the fact that for both a system and a SoS 
that this definition seems to cover both cases given that the definition of SoS is: 

System of Systems: 

A set or arrangement of interdependent systems that are related or 
connected to provide a given capability. The loss of any part of the 
system will significantly degrade the perfonnance or capabilities of the 
whole (Chairman of the Joint Chief of Staff, 2007). 

The literature review revealed that there is a sense that there are different types of 
SoS. Maier and Rechtin proposed that there are three types of SoS and the differences 
are driven by managerial control: Virtual, Voluntary, and Directed (definitions repeated 
here for the ease of the reader). 

• Directed: Directed systems are those in which the integrated SoS is built 
and managed to fulfill specific purposes. It is centrally managed during 
long term operation to continue to fulfill those purposes, and any new ones 
the system owners may wish to address. The component systems maintain 
an ability to operate independently, but their normal operational mode is 
subordinated to the central managed purpose. For example, an integrated 
air defense network is usually centrally managed to defend a region 
against enemy systems, although its component systems may operate 
independently. 

• Collaborative: Collaborative systems are distinct from directed systems in 
that the central management organization does not have coercive power to 
run the system. The component systems must, more or less, voluntarily 
collaborate to fulfill the agreed upon central purposes. The Internet is a 
collaborative system. The IETF works out standards, but has no power to 
enforce them. Agreements among the central players on service provision 
and rejection provide what enforcement mechanism there is to maintain 
standards. The Internet began as a directed system, controlled by the 
Advanced Research Projects Agency, to share computer resources. Over 
time it has evolved from central control through unplanned collaborative 
mechanisms. 
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• Virtual: Virtual systems lack a central management authority. Indeed, 
they lack a centrally agreed upon purpose for the SoS. Large scale 
behavior emerges, and may be desirable, but the supersystem must rely 
upon relatively invisible mechanisms to maintain it. A virtual system may 
be deliberate or accidental. Familiar examples of what is called here a 
virtual system are the World Wide Web and national economies. Both 
‘systems’ are distributed physically and managerially. The World Wide 
Web is even more distributed than the Internet in that no agency ever 
exerted real central control. Control has been exerted only through the 
publication of standards for resource naming, navigation, and document 
structure. Web sites choose to obey the standards or not at their own 
discretion. The system is controlled by the forces that make cooperation 
and compliance to the core standards. The standards do not evolve in a 
controlled way; rather they emerge from the market success of various 
innovators. National economies and the social ‘systems’ that surround us 
might be thought of as virtual systems. Politicians regularly try to architect 
these systems, sometimes through forceful means, but the long-term 
nature is determined by highly distributed, partially invisible mechanisms. 

The TRA Deskbook proposes the following types of IT systems (definitions 
repeated here for the ease of the reader). 

• Business systems - off-the-shelf information system components and 
COTS software assembled together in a new environment to support the 
business and management functions of an organization 

• Net-reliant (battle management) systems - typically command and control; 
battle management systems; or intelligence, surveillance, and 
reconnaissance systems. The net-reliant system is characterized by an 
intense real-time requirement 

• Network infrastructure (or services provider) - backbone and services 
systems for network management, management of large databases and 
glue logic to execute and retrieve services across a Wide Area Network of 
varying security levels. 

• Embedded systems - functionality is enabled by IT but not driven by IT 
itself. Embedded systems emphasize using computer hardware and 
software to automate internal functions of a weapon system such as 
platform control and status, sensor signal and data processing, and 
weapons tasking. 

The following SoS types are proposed. 

• SoS Service Oriented Architecture (SOA) - this is a combination of 
Virtual, Business systems, and Net-reliant giving the sense that a system 
can join the SoS if compliant with it’s standards and protocols in order to 
obtain or supply services. 
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• SoS Common Operating Environment (COE) - this is a combination of 
Directed, Net reliant and Network management since in DoD these are 
typically combined and controlled to provide functionality to a SoS via a 
network. The level of connectedness is engineered for a specific SoS 
rather than generic; it may include common applications but not common 
distributed processing. 

• SoS Common Distributed Processing (CDP) - this a combination of 
Directed and an embedded system approach; embedded doesn’t seem to 
capture the idea of distribution of the processing by all elements of the 
SoS simultaneously. The level of connectedness is at a common 
processing (data/algorithm) level vice a common application level. It 
most likely will include elements of network and network management to 
accomplish the common distributed processing 

The following Family of Systems definition is selected from CJCSI 3170 and the 

Defense Acquisition Guide (Chapter 4.2.6): 

Family of Systems: 

...a set of systems that provide similar capabilities through different 
approaches to achieve similar or complementary effects (Chairman of the 
Joint Chief of Staff, 2007),” and “A family of systems does not create 
capability beyond the additive sum of the individual capabilities of its 
member systems (USD(AT&L), 2006). 

This thesis is primarily focusing on real-time warfighting systems and will follow 
up on SoS SOA and FoS at a future date. 

The definition of SoS gives rise to the idea of systems that are assembled together 
within a boundary to provide a specified capability. This implies that the systems need to 
be able to interoperate in a pre-defined manner in order to be able to predict that the 
‘whole’ will deliver the required capability. 

C. INTEROPERABILITY TAXONOMY 

Interoperability is another word that has no consistent tenninology or use as found 
by the literature review; however, articulating basic types of or intents of communication 
facilities the ability to discuss the degree or types of interoperability required. 
Communication is a broadcast and/or two/multiple way exchange of data/infonnation; 
intent of the communications will drive the types of data/information/knowledge 
exchanges and the timing of same. 
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For DoD interoperability, the assumption is made that communications or 
interoperability is supporting the accomplishment of a task or mission as specified in the 
DoD definition of interoperability. SEI’s definition of interoperability is selected for use 
in assisting in defining degrees of interoperability: 

The ability of a collection of communicating entities to (a) share specified 
information and (b) operate on that information according to a shared 
operational semantics in order to achieve a specified purpose in a given 
context. 

In the literature review it was found that communication consists of the following 
elements: content ( what type of things are communicated), source (by whom), form (in 
which form), channel ( through which medium), destination/receiver (to whom) and 
purpose aspect (with what kind of desired results). Also, it was found that the purpose of 
communications with respect to tasks generally falls into the following categories of 
accomplishing a task or mission: Contribute - to supply, Coordinate - to bring into a 
common action, movement, or condition, Cooperate - to act or work with another or 
others: act together or in compliance, Collaborate - to work jointly with others or 
together, Direct - to regulate the activities or course of. 

The following expanded definitions and associated attributes for these 
communication types are proposed to support defining degrees of interoperability for 
military operations: 

a) Contribute - Provide data/information that supports situational awareness. 
[Independent function] Data/Information is provided when processed by 
the system and receiving system is available and able to receive the 
data/information; no timeliness is associated with contributing 
data/information that is not specifically tied to a mission. 

b) Coordination - determining how and when to share resources for differing 
tasks. Two or more systems execute tasks that are independent from each 
other and require use of the same resources. These differing tasks could 
be done without coordination if there were enough resources; however, 
coordination is required when there are not enough resources to do both 
tasks simultaneously. [Independent function] Data/information is 
provided as requested or required and is the minimum required to indicate 
the need, timelines etc. 

c) Cooperation - the action of multiple systems working together on a 
mission by accomplishing separate and distinct tasks. [Independent (loose 
coupling) - systems are independent and are allocated differing tasks that 
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are required for a mission and so performance of the mission is dependent 
on all systems, may have emergent behavior given the interdependencies, 
bounded system that can be extended.] Data/infonnation is provided in a 
timely fashion and includes data elements that are shared within the 
context of the common mission. 

d) Collaboration - the action of at least two differing systems working 
together on one task by accomplishing similar actions to accomplish the 
task. [Interdependent (tight coupling/synchronized) - Note systems are 
independent and are collaborating on one task that depends on all systems 
to accomplish work towards a common task in order to meet performance, 
bounded system, likely will have emergent behavior given the functional 
dependencies.] Data/information is provided in real-time to near real-time 
and includes multiple data elements that are shared within the context of 
the task and the common mission. 

e) Command and Control - Directive - one system tells the other system(s) 
what to do, when, how, etc. Timelines can be real time (do it now) to non- 
real time (planning). [Independent - Note systems may be providing for 
local or remote command and/or control of resources or actions] 
Data/infonnation is provided in real-time to non-real time and is date 
elements are small in number. 

Degrees of Interoperability based on these definitions are found in Table 5. These 
basic communication types drive the amount, specific data/information elements, data 
flows and timeliness of the communications. The amount of data and the temporal 
aspects of the communication generally are minimal for independent systems, greater for 
systems that are interdependent, and greatest for systems that are interdependent on one 


another to provide a capability. 


Attribute 

Contribute 

Coordinate 

Cooperate 

Collaborate 

Command and 

Control 

Coupling 

None 

None 

Loose 

Tight 

Varies 

Timeliness 

None Req 

Not Req 

Near Real Time 

Real Time 

Varies 

System Type 

Independent 

Independent 

Independent 

Interdependent 

Independent 

Data/Info Type 

Infonnation 

Information 

Data/Info 

Data 

Information 

# of data 

elements 

Small 

Small 

Small/medium 

Large 

Small 


Table 5. Degrees of Interoperability 
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The SEI’s LISI model proposes the following interoperability taxonomy: Isolated 
- non-connected with manual inputs, Connected - electronic connection with separate 
data and applications using homogeneous data exchange mechanisms, Functional - 
minimal common functions with separate data and applications using heterogeneous data 
exchange for basic collaboration, Domain - shared data with separate applications using 
shared databases, and Enterprise - interactive manipulation with shared data and 
applications using automated distributed information exchange applications. 


Alberts et al. Networking the Force Mental Model (seen here again in Figure 26) 
provides for a functional view of communications. Table 6 compares and contrasts the 
different interoperability mappings. 


Networking the Force 


Collection/Analysis 


Sharing 

Awareness 



Execution 

Decisionmaking C2 

1 



Collaboration 


Shared 

Awareness 


Synchronization 


Figure 25. Networking the Force Mental Model [From: Alberts et al., 2001] 
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1MPARISON OF INTEROPERABILITY MODELS 

Communication 

Model 

LISI 

Alberts et al. Mental 
Model 

OSI 

Command and 
Control 

- 

Decisionmaking/C2 
(Shared Awareness) 

- 

Collaborate 

Enterprise 

Decisionmaking/C2 
(Shared Awareness) 
Execution 
(Synchronization) 

Application, 
Presentation, Session 

Cooperate 

Domain 

Sharing (Shared 
Awareness) 

Application, 
Presentation, Session 

Coordinate 

Functional 

Sharing (Shared 
Awareness) 

Transport, Network, 
Data Link, Physical 

Contribute 

Connected 

Collect (Awareness) 

Transport, Network, 
Data Link,Physical 

- 

Not 

Connected 

- 

- 


Table 6. Comparison of Interoperability Models 

For this thesis, the following degrees of interoperability as defined above will be 
used for the analysis to assist in defining the operational relevant environment and for 
enabling technology location and identification. Command and Control will be treated in 
the context in which it is used; for example if it is used for real-time execution of 
operations, it will be treated as Level 4. 


Degrees of Interoperability: Level 0 - Connectionless (self explanatory) 

Level 1 - Contribute 
Level 2 - Coordinate 
Level 3 - Cooperate 
Level 4 - Collaborate 

D. SYSTEMS OF SYSTEMS TECHNOLOGY READINESS ASSESSMENT 
REQUIREMENT/GUIDELINES 

The TRA Deskbook provides a good set of system TRA requirements and 
guidelines. There are no specific requirements or guidance regarding SoS in the TRA 
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Deskbook. Given a SoS is a system that is a set of systems, the current system TRA 
guidance can be easily extended. The following guidelines are recommended with 
respective to SoS TRAs in addition to the current system guidelines. 

1. Clearly describe the type of SoS and degree of interoperability required - 
(SOA, COE, or CDP) and provide rationale. 

2. Indicate which if any of the systems of the SoS is part of another SoS or 
FoS (name related programs). 

3. Identify SoS spirals/blocks or other expected increments and their 
timeframes including spirals/blocks of specific systems of the SoS. 
Provide list of expected changes in architecture, performance, 
functionality and technology. 

4. In the SoS TRA include all CTEs required to meet SoS KPPs/operational 
requirements; include SoS unique CTEs as well as system unique CTEs 
required for the specific system to participate as a system of the SoS (e.g., 
a new radio) regardless of who is responsible for funding or developing. 

5. Provide an update to the SoS TRA when any of the systems of the SoS are 
going thru a spiral upgrade independently of the SoS. Each system of the 
SoS needs to be assessed for any changed technology or technology 
implementation to assure SoS perfonnance is preserved. 

6. SoS Milestones B and C shall be scheduled post system Milestone Bs and 
Milestone C’s for SoS (or at least in the same timeframe, +/- 3 months). 
Specific systems need to demonstrate TRL 6 and TRL 7 prior to 
demonstrating SoS TRL 6 and SoS TRL 7. 

7. Systems that are part of a SoS shall include SoS CTEs in their system 
(SoS in the case of a IAMD SoS) specific TRA. Each individual system 
will need to develop their system specific technologies to a TRL 6 and 
above as well as demonstrate system functionality with SoS specific 
technologies to a TRL 6 and above. 

8. All SoS CTEs whether part of the SoS or part of the specific systems of 
the SoS, shall be assessed against SoS requirements. These assessments 
should begin as early as possible and should begin at TRL 3 assessments. 

If a Program Review is used to initiate the SoS, direction shall be provided to 
systems of the SoS to begin SoS Engineering activities including technology 
development and assessment. All SoS CTEs including the system specific CTEs for SoS 
activities need to be matured prior to SoS MS B. The synchronization and technology 
maturation strategy shall be identified in the Acquisition Decision Memorandum and 
defined in the Acquisition Strategy and TDS. 
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E. SOS TECHNOLOGY LOCATOR CHECKLIST 


It is useful to have a checklist when performing a TRA for a SoS given the 
number and complexity of systems that generally make up a SoS. Also, it is a challenge 
to identify where unexpected effects may occur and a checklist will assist in lowering the 
risk of finding these unexpected effects post SoS development. Generally there is a SoS 
PM and PMs for each of the systems that make up a SoS. It is recommended that the 
system program managers and SoS PM coordinate regarding common SoS technologies 
needs as well as system specific technologies that are required to interface with the SoS 
technologies. System technologies may not pass the criteria for being a system CTE; 
however, once a decision is made for the system to be part of the SoS, the CTE in 
question may become a SoS CTE. Some system specific CTEs may not be a SoS CTEs; 
an example is advanced armor for a ta nk that is part of FCS would not be part of the FCS 
SOSCOE CTEs. The technology list should make a distinction between technologies that 
are common to all systems of the SoS and those technologies that are unique to a specific 
system of the SoS that is required in order to work in the SoS operational context. If a 
technology is required to enable a system to meet SoS KPPs/operational requirements it 
should be included in the SoS CTE list. 

A SoS (IT) technology locator/identification checklist is recommended. SoS 
technology identification requires knowledge of the systems in the SoS, the types of 
functions and computational processes being performed cooperatively and collaboratively 
(horizontally and vertically) and the specific functionality and required behavior and 
performance. Documentation should indicate by what methods these requirements will 
be achieved: 1) procedures - doctrine, mission, architectures, and standards, 2) 
applications and 3) hardware/software or a combination thereof. Specific 
technologies/category of technology should be noted if known. The following 
interoperability/synchronization related attributes should be used as lines of inquiry when 
executing the analysis with respect to each facets of the environment - physical, logical, 
data, security and user. 

Interoperability Attribute List: 

• Completeness - all relevant items available, including entities, their 
attributes, and relationships between them 
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• Correctness - all items in the system faithful representations of the realities 
they describe 

• Currency - latency of the items of information 

• Accuracy or Level of Precision - dependent on the purpose 

• Consistency - across different systems, applications, functions and 
data/information/knowledge. Note Data models that the SoS is expected 
to be in compliance with. 

• Connectivity - specified integration of nodes, type of connections, 
syntactic compatibility, quality of service and bandwidth/data rate 
requirements 

• Capacity - databases, scalability, number and type of applications, 
processor requirements 

• COTS - use of and consideration of obsolescence, instability in standards 
or availability, security, and reliability 

The technology locator/identification checklist should be used to identify all 
technologies (not just CTEs) in both the SoS and the systems that are part of the SoS (if 
the particular artifact is not available to the program, a similar design or requirements 
document should be identified and used). Also, this should be accomplished for each 
spiral or block of each system and the SoS. If new technologies are identified to meet the 
requirement and they are not achievable in the timeframe due to unavailability of 
technologist or there is not enough funding to mature them, this should be noted and a 
recommended adjustment in requirements and technology solutions propose to the PM. 
After all technologies have been identified and appropriate technology alternatives the 
criteria for CTEs can be applied: 

1) An approved ICD or draft/approved Capability Description Document 
whichever is latest: this provides the KPPs/operational requirements. 

a. Evaluate and document how much of an increase or change in 
capability is being required from currently fielded systems. 

b. Determine and document what current technologies can be 
modified to meet the new requirements or if the change in 
capability requires fundamentally new technologies, describe what 
technologies would be required if known. 

2) An OV-1; this provides the operational context and concept. 

a. Document the systems that are in the SoS and what requirements 
they are now meeting wrt the KPPs and with what technologies 
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b. Document which systems have worked together in a SoS before or 
currently and to what ends and what technologies were in common 
(that are applicable to this specific SoS). 

3) An approved/draft concept of employment. 

a. Document any major difference between current operations and 
expected approved/new operations. 

b. Document the type of technologies that would be required to 
enable the new concepts. 

4) An approved/draft OV-2 and an approved/draft SV-1 and SV-2; this 
identifies the specific operational nodes/systems, the operational activities 
at each node, and the information exchanges and interconnections needed 
between nodes. 

a. Document the connectivity requirements required to enable these 
activities from the interoperability attribute list above (e.g., 
bandwidth, types of networks/datalinks) and identify technology 
requirements. 

b. Document the data/information/knowledge exchanges required and 
identify technology requirements. 

5) An appro ved/draft OV-3 and SV-3; identifies the interfaces and the 
information exchanges between nodes. Document any new/unusual 
(special) interfaces or information exchanges that would require new 
algorithms, methods or techniques to process and/or encode/decode 
data/information, (document proposed technologies). 

6) An appro ved/draft OV-5; identifies capabilities, relationships among 
activities and inputs and outputs 

a. Document new/modified required capabilities and any 
new/modified relationships and the technologies needed to enable 
these. 

b. Document new/modified input/outputs and the technologies 
required that would enable networking, network management, or 
new processing techniques to meet these capabilities. 

7) An appro ved/draft OV-6; describes the sequencing and timing of activities 
as well as business rules and processes. An executable simulation is 
highly desired. 

a. Document timing requirements and note those that are more 
stressing than that currently fielded; propose technologies that 
would be required to meet these. 

b. Document new types of business rules and processes that require 
new/modified algorithms, techniques and methods - note where 
there is an expectation of common processing of data across an 
interface; identify the technologies required. 
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8) An approved/draft OV-7; documents the data requirements and business 
rules. Document any special data requirements, business rules and the 
algorithms, techniques and methods required. 

9) An appro ved/draft SV-4 and SV-5; The SV-4 documents the system 
functions and the data flow between them and the operational activities 
supported 

a. Document all major system functions, the operational activities 
associated with them, the data flows between them and the 
technologies required to enable these functions. Note those 
functions that are required for synchronization. 

b. Document all expected interacting technologies and/or have 
dependencies on other technologies. 

10) An appro ved/draft SV-6 and SV-11 documents the data element 
exchanges and the physical implementation e.g., messages. Document 
any technology requirements to enable the data flow and implementation 
in a physical architecture. 

11) An approved/draft SV-7; documents the performance characteristics 
including the timelines. Document technologies and their requirements to 
meet functional, behavioral and performance requirements 

12) An appro ved/draft TV-1 and TV-2 - current and future standards; 
document which standards are required for the systems and identify 
new/modi lied technologies that are required. 

13) Review proposed list of new/modified technologies/subsystem/systems to 
determine if technology may impact the following or may be limited in its 
use; identify possible technology options that will not adversely impact 
operational resources: 

a. Manning impacts: In-theater, reach-back capabilities, Knowledge 
/Skills/Ability requirements changes for current billets. 

b. New data requirements, new security/information assurance 

requirements and technologies required to meet these 
requirements. 

c. New/Modifications in operational procedures (Service, Joint, 
Coalition). 

d. New/Modifications in logistic or other support requirements (e.g 
calibration resources, batteries). 

e. New/Modification in operational planning. 

f. New/Modifications in IT support/databases/network equipment 

etc. 

g. New/Modi (ications in training. 

h. Adversely impact operational budgets. 
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i. Require legal or other policy changes including impacts on the 
environment (overseas laws and standards may differ from those of 
the United States). 

j. Impact health and/or safety. 

Figure 27 provides a good summary of the analysis that this checklist embodies. 
Ideally this analysis is performed collaboratively between the S&T and acquisition teams 
supporting the SoS Engineering activities. At the end of the analysis one should have a 
complete list of new/modified/current technologies and/or category of technologies being 
used, planned for use, or require development in order to achieve the SoS KPPs mapped 
to the operational architecture, system architecture and physical architectures. 

This list can then be vetted against the CTE criteria found in the TRA Deskbook 
to determine the CTEs that should be addressed in a TRA. 

F. SELECTED DOD PROGRAM ANALYSIS 

Four DoD programs are analyzed with respect to the research questions. 

1. Theater Battle Management Command System 

From the literature review, TBMCS is a set of applications used to collect, 
process and distribute data in support of the AOC. The AOC has been described as a SoS 
or a complex system created from an opportunistic aggregation of systems (80+ 
applications and systems) and has a sense of unboundedness. (See Figure 28 for the set of 
AOC applications) No ‘two’ AOCs are the same with any probability. Initially it was 
intended to integrate the functions of three systems major systems: CTAPS, which was 
under development, the Wing Command and Control System, and the Combat 
Intelligence System. At the start of the program TBMCS did not have an Operational 
Requirements Document/CDD or a Concept of Operations on how it was to be used in 
the field. The system architecture was defined at a high level. These factors made it 
impossible to test with any established criteria. Testing that did occur did not exercise 
concurrent processing and the first tests failed (Collens Jr., 2005). 
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Figure 26. Using Architecture in System Engineering [From: Dickerson and Soules, 

2002] 

Eventually the TBMCS program established high level requirements and put in 
place a process to establish system requirements and engineering processes. TBMCS 
was directed to use DII COE. TBMCS has evolved from a large client-server application 
to a much more streamlined, web-based enterprise over the last few years. These actions 
provide a way for the program to progress more successfully. 

The following figures show the operational and system architectures for TBMCS. 
Figure 29 - TBMCS Notional Theater C4I, Figure 30 - TBMCS Functional Description, 
Figure 31 - TBMCS interfaces, Figure 32 - TBMCS with DII COE, Figure 33 - TBMCS 
Communication Architecture and Figure 34 TBMCS Data Architecture. 
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Figure 27. AOC System List [From: Norman and Kuras, 2004] 
a. TBMCS SoS or Not? 

TBMCS is assessed to be a collection of systems facilitated by a COE (DII 
COE), but are not interdependent (not a SoS) on each other to accomplish their specific 
tasks and not a FoS that would have differing systems to accomplish the same mission. It 
may be useful to call it an Enterprise (a systematic purposeful activity (Merriam-Webster, 
2007). 
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Figure 28. Notional Theater C4I [From: Collens Jr. and Krause, 2005] 



Figure 29. TBMCS Functional Description [From: Collens Jr. and Krause, 2005] 
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b. TBMCS Interoperability 

It operates with other systems via contribution (Level 1), coordination 
(Level 2) and cooperation (Level 3) and appears to operate as an Enterprise. It is not 
evident TBMCS is driven by real-time collaboration of simultaneous work on the same 
task; Cooperation regarding a mission is facilitated by the COE but not necessarily 
enabled by it. The TBMCS has its own database and added functionality to the DII COE 
for various applications. 
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Figure 30. TBMCS Interfaces VI. 1.3 [From: Collens Jr. and Krause, 2005] 


The most important interoperability attributes for TBMCS are: 

• Completeness - all relevant items available, including entities, their 
attributes, and relationships between them 

• Correctness - all items in the system faithful representations of the realities 
they describe 

• Accuracy or Level of Precision - dependent on the purpose 

• Connectivity - specified integration of nodes, type of connections, 
syntactic compatibility, quality of service and bandwidth/data rate 
requirements 
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• Capacity - databases, scalability, number and type of applications, 
processor requirements 

• COTS - use of and consideration of obsolescence, instability in standards 
or availability, security, and reliability 

The other attributes while important, were not drivers for CTEs. 

c. TBMCS TRA Requirements and Guidelines 

TBMCS has defined spirals with the emphasis on all systems using the DII 
COE. Systems are expected and encouraged to bring themselves into the TBMCS 
‘enterprise’. It’s unclear what coordination or engineering occurs between the TBMCS 
and system PMs. 

There was no formal declaration of TBMCS as a SoS or FoS program. 
Synchronization of specific applications or programs is not evident from a planning 
perspective. It’s unclear that a TRA for TBMCS was considered; applications or systems 
are required to mature their own systems. It is u nkn own whether system TRAs included 
the DII COE elements in their TRA. 



Figure 31. TBMCS with DII COE II Architecture [From: Collens Jr. and Krause, 2005] 
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Figure 32. TBMCS Communications Architecture [From: Collens Jr. and Krause, 
2005] 
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Figure 33. TBMCS Data Architecture [From: Collens Jr. and Krause, 2005] 
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d. 


TBMCS CTE Identification 


TBMCS did not have the usual requirements, architecture and system 
engineering artifacts. Its unclear whether there were any CTEs identified. Several 
standards were directed - DII COE and it inherited the differing standards from the 
legacy systems. TBMCS in it’s first rendition was understood by inspection TBMCS was 
found to have four types of integration which appear to support the degrees of 
interoperability identified above: 1) internal interfaces and subcomponents, 2) varying 
level of maturing third party applications, 3) external interfaces, and 4) databases. Fully 
90 percent of TBMCS consisted of third-party products or government-furnished 
equipment (GFE), and a majority of the software was third-party: GOTS or COTS. 
TBMCS incorporated 76 applications, 64 point-to-point external system interfaces, and 
413 segments involving over 5 million lines of software, as well as two commercial 
relational databases (Collens Jr., 2005). The system had two hardware baselines, and the 
communications infrastructure the DII COE. The most extensive integration involved 
data interoperability, and the two primary TBMCS databases - the Air Operations Data 
Base and the Intelligence Server Data System - followed different standards and were 
updated at different intervals. The government also mandated the use of specific 
hardware, which varied depending on the service branch that would use TBMCS. A 
particular application requested by the user might not integrate well into the system 
because it did not use the DII COE properly or because its COTS infrastructure was more 
current than that of TBMCS. 

TBMCS applications are now migrating from a client-server system to 
web-based architecture over various spirals. A TBMCS Developer’s Network 
(DEVNET), a collaborative effort of the Electronic Systems Center (ESC) and Lockheed 
Martin, enables third-party Air Operations Center (AOC) system developers to easily 
integrate new applications into TBMCS in a non-proprietary, ‘plug and play’ open 
architecture environment. 
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e. 


TBMCS Technical Challenges 


TBMCS seems to have experienced all the technical challenges a program 
could possibly encounter. There were no system engineering efforts initially. After high 
level requirements were established, capability requirements and functional analysis 
occurred at the same time as system requirements, system functional analysis, and system 
technology development were being accomplished; however, TBMCS was being 
assembled from legacy systems. Only over time has TBMCS been able to start to 
overcome these historical hurdles and move towards network-centric functionality. 

It’s unclear that today TBMCS high level performance parameters are 
allocated to individual systems and their subsystems. TBMCS is changing constantly 
over time and is enabled with varying different hardware. 

/. TBMCS Systemic Challenges 

Critical technologies developed by the individual programs don’t appear 
to be in any particular alignment with an overall TBMCS schedule. No overarching 
technology development strategy seems evident. It’s unclear what milestones TBMCS 
and all its systems need to pass through. Testing is challenging, there appear to be 
selected opportunities for integrated testing. 

The fielding of TBMCS capability is time-phased over several spiral 
upgrades several years apart with a focus on becoming totally web-enabled with systems 
being able to be incorporated via the DII COE. Program synchronization seems 
unachievable based on the all-encompassing nature of the program; however, over-time 
stability of individual applications or systems should increase. This makes sense in the 
context of an Enterprise. 

2. Future Combat System (FCS) 

The Army's Future Combat Systems (FCS) currently includes 14 elements plus 
the network and the soldier (See Figure 35). The network allows the FCS Family-of- 
Systems (FoS) to operate as a cohesive SoS where the whole of its capabilities is greater 
than the sum of its parts. The FCS program anticipates needing to interoperate or 

integrate with as many as 170 systems, some of which are in development and many are 
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legacy. Many complementary programs are not being developed exclusively for FCS and 
are outside the direct control of the FCS program, such as their communications 
networks. 


The FCS network consists of four overarching building blocks: System-of- 
Systems Common Operating Environment (SOSCOE); Battle Command (BC) software; 
communications and computers (CC); and intelligence, reconnaissance and surveillance 
(ISR) systems. SOSCOE is central to the FCS network, which supports multiple 
mission-critical applications independently and simultaneously. It is configurable so that 
any specific instantiation can incorporate only the components that are needed for that 
instantiation. SOSCOE architecture uses COTS hardware and a DISR compliant 
operating environment to produce an non-proprietary, standards-based component 
architecture for real-time, near-real-time, and non-real time applications (Figure 36) 
(Future Combat System Program Office, 2007). The following figures show the FCS 
operational, system, and physical architectures: Figure 37 - FCS OV-1, Figure 38 - FCS 
Communications Architecture, Figure 39 - Network Concept of Operations, Figure 40 - 
FCS System Architecture, Figure 41 - SoS Approach, and Figure 42 - LANDWARNET. 
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Figure 35. FCS Enabled by SOSCOE [From: Child, 2006] 
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Figure 37. FCS Communications Architecture [From: Future Combat System Program 
Office, 2005] 
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Figure 38. FCS Network Concept of Operations [From: Future Combat System 
Program Office, 2005] 
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Figure 39. FCS System Architecture [From: 36 Future Combat System Program Office 
2005] 
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Figure 40. FCS SoS Approach [From: Child, 2006] 
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Figure 41. FCS LANDWARNET [From: Future Combat System Program Office, 
2007] 


a. FCS SoS or Not? 

FCS is assessed to be a FoS enabled by FCS SOSCOE. The FCS program 
is not required to meet performance beyond the capabilities of the individual systems and 
there seems to be no intent of interdependencies. The SOSCOE is used to provide 
commonality in communications. 

b. FCS Interoperability 

It operates with other systems via contribution (Level 1), coordination 
(Level 2) and cooperation (Level 3) similarly to today’s systems. The SOSCOE 
facilitates these processes by providing standardization for these processes. 

The most important interoperability attributes for FCS are: 

• Completeness - all relevant items available, including entities, their 

attributes, and relationships between them 
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• Correctness - all items in the system faithful representations of the realities 
they describe 

• Accuracy or Level of Precision - dependent on the purpose 

• Consistency - across different systems and applications (tailored) 

• Connectivity - specified integration of nodes, type of connections, 
syntactic compatibility, quality of service and bandwidth/data rate 
requirements 

• Capacity - databases, scalability, number and type of applications, 
processor requirements 

• COTS - use of and consideration of obsolescence, instability in standards 
or availability, security, and reliability 

The other attributes are not drivers of CTEs. 
c. FCS TRA Requirements and Guidelines 

FCS identified their FoS enabled by a SOSCOE up front. They have 
identified spirals (see Figure 43) and adjustments have been made over time. 
Unfortunately only 18 of the 49 technologies currently rated have demonstrated TRL 6 
and none of the critical technologies may reach TRL 7 until the production decision in 
fiscal year 2012. Given this is a FoS; only those COE or infrastructure CTE would 
impact the FCS in the short term; as the FCS should be able to be fielded as systems are 
ready once a SOSCOE is ready. In the GAO report 06-367 (GAO, 2006) there are 13 
related SOCOE networking technologies. The rest are associated with specific functions 
or systems. 

Technology maturation plans have been delayed or not kept up with 
predicted maturation expectations by 3-5 years in some cases. The Critical Design 
Review (CDR) is being held two years prior to production; this seems very unrealistic 
given the advanced hardware technologies in development. The program structure seems 
to be based on calendar dates not knowledge points (See Figure 44). FCS development 
status has been the subject of at least two GAO reports for the lack of mature 
technologies and program structure issues. 
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Figure 42. FCS Acquisition Approach [From: Future Combat System Program Office, 
2005] 

d. FCS CTE Identification 


FCS identified 49 CTEs; however, they may not have identified all of 
them given they have 550 operational requirements and upwards of 11,500 so called SoS 
level requirements and it is anticipated to have upwards of 90,000 system requirements 
(GAO, 2006). Requirements are not stable and so by definition the architecture, 
technology selections may change. This may be mitigated given performance is allowed 
to be just as good as previous systems not connected via a common network. 


The FCS CTEs list is not inclusive of all the programs required by FCS, a 
total of 52 programs and their associated technologies have been determined to be critical 
in FCS meeting their required KPPs. Currently, synchronization of schedules is not an 
option due to the other systems own challenges. 
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Best practices approach 


Development Production 

start start 



2000 2002 2004 2006 2008 2010 2012 2014 2016 

KP 1: (Knowledge Point 1): technologies and resources match requirements 
KP 2; (Knowledge Point 2): design performs as expected 

KP 3: (Knowledge Point 3): production can meet cost, schedule, and quality targets 
PDR: Preliminary design review 
CDR: Critical design review 

Figure 43. FCS approach to integration of COTS technologies [From: GAO, 2006] 

e. FCS Technical Challenges 

FCS is building new systems. Capability requirements and functional 
analysis should occur prior to specific system requirements, system functional analysis, 
and system technology development. They continue to experience system requirements 
chum even though they are past the System Functional Review. They have spent a lot of 
time and money to determine allocations of requirements to systems; however, that work 
is not finished. 

FCS is developing a robust modeling and simulation capability; however, 
with a new type of system (SOSCOE in this case) only a test with the real SOSCOE will 
show what really happens when used to connect the FoS. It is a definite danger that all 
CTEs may not been identified given the heterogeneous nature of FCS. 

/. FCS Systemic Challenges 

FCS is experiencing and will continue to experience a high level of churn 
given there are more systems outside the FCS umbrella (52 systems) than inside the FCS 
umbrella (14+1+1) that required to meet the FCS KPPs. The lack of simultaneous system 
engineering activities, aligned schedules and detailed visibility into the requirements and 
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development and testing of the 52 external essential systems is scary! A time-phased 
rollout of FCS that is aligned with the maturation and delivery of the external systems is 
essential. It’s unclear whether the first spiral has included more than can be managed. 

3. Army’s Integrated Air Missile Defense Systems of Systems (AIAMD 
SoS) 

Army’s Integrated Air and Missile Defense (IAMD) System of Systems is a 
capability designed to be deployed as an integrated set of system interoperable and able 
to synchronize operations with Army, Joint, Interagency, Intergovernmental and Multi- 
National (JIIM) Net-Centric architectures and systems. The IAMD Battle Command 
System (IBCS) is designed to be the integrator. (See Figure 45) Common applications 
via Plug and fight modules are used to interface to legacy sensors, control, and 
engagement systems (See Figure 46 and 47). The SIAP IABM is part of these plug and 
fight modules (See next section). The Army adds their service specific common 
functionality to the plug and fight modules as well. IAMD SoS products are shown in 
Figure 48. 


a. AIAMD SOS, SoS or Not? 

FCS is assessed to be a SoS CDP given they are enabled by common 
processing from SIAP in from their plug-n-fight modules. 



Figure 44. AIAMD SOS Architecture [From: IAMD Program Office, 2007] 
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b. AIAMD SOS Interoperability 

It operates with other external systems via contribution (Level 1), 
coordination (Level 2), cooperation (Level 3) and collaboration (Level 4) among the 
systems of the IAMD SoS and the SIAP SoS. 

Key interoperability Attributes: 

• Completeness - all relevant items available, including entities, their 
attributes, and relationships between them 

• Correctness - all items in the system faithful representations of the realities 
they describe 

• Currency - latency of the items of information 

• Accuracy or Level of Precision - dependent on the purpose 

• Consistency - across different systems, applications, functions and 
data/information/knowledge. Note Data models that the SoS is expected 
to be in compliance with. 

• Connectivity - specified integration of nodes, type of connections, 

syntactic compatibility, quality of service and bandwidth/data rate 
requirements 

• Capacity - databases, scalability, number and type of applications, 

processor requirements 

• COTS - use of and consideration of obsolescence, instability in standards 
or availability, security, and reliability 

c. AIAMD SOS TRA Requirements and Guidelines 

AIAMD clearly identified they are a SoS and have made reference to their 
connection with SIAP. They have identified their detailed schedules (see Figure 49) for 
spiral one and indicated the functionality for future spirals (see Figure 50). 
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• IAMD Common Battle Command Provides: 

- IBCS CPs to Include All Vehicles, Shelters, 
Hardware, Software and Support Equipment 

- IBCS Training Devices and Manuals 

- IBCS CP (-) as Part of JLENS GS 

- IBCS CP (-) as Part of THAAD FC 

• Plug and Fight Interface Modification Kits to 
Include Fire Control Net Radios and IAMD 
Hardware/Software Functionality for*: 

- Patriot Launcher Farm - Improved Sentinel 

- SLAMRAAM Launcher - Patriot Radar 

- MEADS Launcher - ADAM Cell 

• IFC Net Relay (Mod Kits) 


Enterprise Net-Centric Products 
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Figure 45. AIAMD SoS System Approach [From: IAMD Program Office, 2007] 
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Figure 46. AIAMD Plug-n-Fight Interfaces [From: IAMD Program Office, 2007] 
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Figure 47. AIAMD SoS Product End Items [From: IAMD Program Office, 2006] 

Army started the IAMD SoS appropriately and has embarked on 
technology development. They are reliant on other program managers to develop 
technologies such as the Warfighter Integrated Network - Tactical (WIN-T) similarly to 
the FCS program and on SIAP to develop tracking, Combat Identification (CID), and 
network related algorithms for common distributed processing. 

The IAMD SoS has synchronized their internal schedules appropriately; it 
is unclear how successful they can be with respect to external system development 
alignment with systems such as WIN-T which currently has several technology issues. 

AIAMD SoS will be expecting spiral upgrades in short time cycles than 
SIAP SoS. SIAP SoS TRA will need to be updated as these spirals are defined and 
technologies are changed/removed or added. 

d. AIAMD CTE Identification 

AIAMD SOS has done a good job of identifying CTE for AIAMD and the 
intersection with SIAP SoS CTEs. It is unclear whether their SoS CTE list includes other 
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essential SoS technologies provided by other programs. The SIAP SoS System 
Functional Review (SFR) has been held and so risk should be mitigated in this area. 

e. AIAMD SOS Technical Challenges 

AIAMD is experiencing some challenges given they are part of the SIAP 
SoS that provides common distributing processing and rely on the ballistic missile 
defense related technologies from MDA. They have addressed the issue of connecting 
legacy systems together by using a plug-n-fight module and IBCS approach; this 
mitigates changes to existing systems. 



UNCLASSIFIED 1 


Figure 48. AIAMD Acquisition Schedule [From: IAMD Program Office, 2007] 
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Figure 49. AIAMD Spiral Development [From: IAMD Program Office, 2007] 

They will face issues regarding proper allocation of KPPs to IAMD SoS 
from both SIAP SoS and IAMD SoS. They are building a representative modeling and 
simulation environment to include IAMD and SIAP SoS as it related to Army. 

/. AIAMD SOS Systemic Challenges 

AIAMD SOS may have alignment issues with other programs. They have 
not gone to a MS B, therefore, these may be able to be worked prior to that event. They 
are dependent on legacy systems development and integration timelines and may 
experience some delays in fielding AIAMD SoS. AIAMD SOS has laid out a spiral 
acquisition so which can mitigates delays; however, AIAMD SOS systems related to air 
defense likely would have an impact to meeting SIAP SoS KPPs which require common 
distributed process and networking technologies. 

4. Single Integrated Air Picture System of Systems 

Single Integrated Air Picture (SIAP) is the product of fused, common, continuous, 
unambiguous tracks of all airborne objects in the surveillance area. The SIAP OV-1 can 
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be seen in Figure 51. The SIAP is developed from near-real time and real time data, and 
is scalable and filterable to support situation awareness, battle management, and target 
engagements. SIAP is created by the collaboration of multiple systems (see Figure 52 - 
SIAP SoS Boundary). SIAP is accomplished via an Integrated Architecture Behavioral 
Model (IABM) which when instantiated in a combat system provides for distributed 
common processing of data/information (see Figure 53 - SIAP Functional Architecture, 
54 - SIAP IABM as executable specification into SIAP SoS Systems, and Figure 55 - 
SIAP IABM instantiation into SoS system). The IABM is built using a Model Driven 
Architecture™ approach. This mitigates the ambiguity of ‘paper’ specifications which 
leads to divergence of implementation. Using the IABM provides for a common 
distributed process of data for track management, sensor registration, composite tracking 
and combat ID. 


a. SIAP SoS or Not? 

SIAP is assessed to be a SoS CDP enabled by IABM. 

b. SIAP Interoperability 

SIAP is created collaboratively (Level 4) by the systems of the SoS to 
meet SIAP KPPs. Given that SIAP is an enabling capability the other levels of 
interoperability didn’t seem to apply. 

Interoperability Attribute List: 

• Completeness - all relevant items available, including entities, their 
attributes, and relationships between them 

• Correctness - all items in the system faithful representations of the realities 
they describe 

• Currency - latency of the items of information 

• Accuracy or Level of Precision - dependent on SIAP KPPs 

• Consistency - across different systems, applications, functions and 
data/information/knowledge. Note Data models that the SoS is expected 
to be in compliance with. 

• Connectivity - specified integration of nodes, type of connections, 
syntactic compatibility, quality of service and bandwidth/data rate 
requirements 
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• Capacity - scalability (tailored) 

Other interoperability attributes not drivers for the SIAP CTEs. 

c. SIAP TRA Requirements and Guidelines 

SIAP has identified the target systems for instantiation of IABM. The 
IABM can be used by other SoS as shown by AIAMD SoS. All SIAP CTEs including 
system specific CTEs of those systems participating in SIAP, for example AIAMD SoS 
use of WIN-T is required for SIAP are in the SIAP SoS TRA. 

SIAP SoS has not yet had its MS B; however, it is anticipated there will be 
blocks or spirals to add functionality or systems to the SoS. A Joint SoS TRA has been 
accomplished for SIAP. See Figure 55 for the location and types of CTEs. SoS 
Engineering has been occurring with all Services and several service programs. PDR is 
expected to complete in a few months. SoS Test events occur on a regular basis to 
mature the SoS CTEs. 


d. SIAP CTE Identification 

SIAP CTEs were identified by representatives from all the services for the 
SoS unique and specific system CTEs required for participation in SIAP SoS using 
knowledge of the OVs and SVs and other engineering artifact regarding SIAP. As a 
system is added to the SoS, the specific system will be reviewed for their specific system 
CTEs as well. 
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Figure 50. SIAP OV-1 [From: Wilson, 2004] 



Figure 51. SIAP SoS Boundary [After Ref Wilson, 2004] 
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Figure 52. SIAP Functional Architecture [After Ref Wilson, 2004] 



Figure 53. SIAP IABM as executable specification into SIAP SoS Systems [From: 
Wilson, 2004] 
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Figure 54. SIAP IABM Instantiation into a SIAP SoS system [From: Wilson, 2004] 



Figure 55. SIAP CTE Locations [After Ref Wilson, 2004] 


e. SIAP Technical Challenges 

The SIAP KPPs were not finalized until recently and so there was some 
risk of selecting technologies that wouldn’t meet SIAP KPPs; this risk was not realized. 
KPP allocation to specific systems is problematic and analysis will be ongoing. SIAP 
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SoS is made up of a mix of new and legacy systems. Work is ongoing to create 
appropriate modeling and simulation/test venues to cover the diversity of Service 
systems. 


/. SIAP Systemic Challenges 

It has been a challenge to have system specific CTEs that are required to 
meet SIAP KPPs matured in alignment with SIAP SoS schedules. The SIAP SoS was 
originally structured to only mature the IABM CTEs and not all the SoS CTEs. The SoS 
KPPs could not be adequately tested beyond TRL 5 without representative systems 
instantiating the IABM and developing their system specific CTEs for participation in 
SIAP SoS testing (e.g., radios, sensor algorithms). SIAP will be initially fielded with a 
small number of systems and as systems are available they will be added. 
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IV. PRELIMINARY ANALYSIS 


A. APPROACH 

Based on the research questions and the literature review, definitions of SoS and 
types of SoS, a taxonomy for degrees of interoperability, SoS TRA requirements and 
guidelines and a CTE identification checklist were derived from the literature review and 
applied during the research phase to four DoD systems that were identified as potential 
SoS. 

Analysis of the research with respect to the research questions are reviewed and 
analyzed to provide a basis for final conclusions and recommendations regarding SoS 
TRA requirements and guidelines, SoS definitions and/or type definitions, degree of 
interoperability definitions, SoS acquisition guidance, and TRL description 
modifications. 

B. SOS TECHNOLOGY READINESS ASSESSMENTS ANALYSIS 
1. Relevant Environment 

In order to adequately conduct a TRA, the operational relevant environment needs 
to be characterized. An understanding with regard to type of system and the degree of 
interoperability are fundamental to understanding the environment. Given the literature 
review, the proposed system and degree of interoperability definition, four potential SoS 
were analyzed to determine if these definitions were valid. 

a. SoS Definitions and Types 

The research showed thatsystems that may be labeled as SoS do not 
necessarily meet the same SoS definition. The definition of SoS that was used was the 
following: 

System of Systems: 

A set or arrangement of interdependent systems that are related or 
connected to provide a given capability. The loss of any part of the 
system will significantly degrade the perfonnance or capabilities of the 
whole (Chairman of the Joint Chief of Staff, 2007).” 
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It was found that neither TBMCS or FCS qualify as a SoS, rather a 
common COE facilitates communication, contribution and cooperation (albeit different 
COEs). 

It was found that AIAMD and SIAP do qualify as a SoSs. These two 
systems use common distributed processing via embedded algorithms from both SIAP 
and the IAMD SoS in the plug-n-fight modules and the integrating IBCS. In addition, the 
SIAP SoS is not predicated on having a COE across the SoS; therefore, just having a 
COE isn’t a detennining factor of whether a program is a FoS, SoS, or other type of 
system. 

There was not enough research perfonned to determine whether the 
proposed types of SoS - SOA, COE, and CDP are valid. It was found in this research 
that having a COE is more of an infrastructure or backbone for communications rather 
than indicator of collaborative common distributive processing. The FCS program may 
intend in the future to require synergistic perfonnance increased and include this in their 
COE. Further research will be required to determine whether SoS SOA or SoS COE are 
valid specific types of SoS. 

Given the definition of SoS and detennining what systems are SoS 
provides the basis for level and type of system engineering and program acquisition 
planning. 


b. Degrees of Interoperability and Interoperability Attributes 

The research showed that differing types of systems have different types 
of interoperability. The definition determined via the literature review and used during 
the research showed a continuum of interoperability from connectionless through 
collaborative. 

Degrees of Interoperability: Level 0 Connectionless (self explanatory) 

Level 1 Contribute 

Level 2 Coordinate 
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Level 3 Cooperate 

Level 4 Collaborate 

The literature review and research showed that the term collaboration is 
used colloquially by programs to mean systems are connected in some way without 
making a distinction as to purpose. 

It was found that TBMCS, FCS, and AIAMD SoS all interoperate and 
perform contribution, coordination and cooperative actions. SIAP SoS was found to 
mainly support collaboration activities. Given this, it appears useful to designate the 
degrees of interoperability required rather than defaulting that the highest level of 
interoperability is characterized the SoS. 

The definitions of interoperability facilitated verification of the system 
type (FoS, SoS, or other) and prepped the research for technology location/identification. 
It was expected that for each of the interoperability attributes that the degree of 
interoperability would drive these behaviors. Each system was evaluated with respect to 
the Interoperability Attribute List (repeated here): 

• Completeness - all relevant items available, including entities, their 
attributes, and relationships between them 

• Correctness - all items in the system faithful representations of the realities 
they describe 

• Currency - latency of the items of information 

• Accuracy or Level of Precision - dependent on the purpose 

• Consistency - across different systems, applications, functions and 
data/information/knowledge. Note Data models that the SoS is expected 
to be in compliance with 

• Connectivity - specified integration of nodes, type of connections, 

syntactic compatibility, quality of service and bandwidth/data rate 
requirements 

• Capacity - databases, scalability, number and type of applications, 

processor requirements 

• COTS - use of and consideration of obsolescence, instability in standards 
or availability, security, and reliability 
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The research showed the following: 

The TBMCS and FCS were not as concerned with currency and 
consistency as compared with AIAMD SOS and SIAP. Currency and consistency are the 
key drivers of CTEs for collaborative common distributive processing. In generally all 
the interoperability attributes are of interest to any system that is communicating with 
others if cooperating to accomplish a mission or collaborating on the same task. 

2. SoS TRAs 

a. SoS TRA Requirements and Guidelines 

The following SoS TRA requirements and guidelines were developed 
from the literature review and used during the analysis of the four potential SoS: 

1. Clearly describe the type of SoS and degree of interoperability required - 
(SOA, COE, or CDP) and provide rationale. 

2. Indicate which if any of the systems of the SoS is part of another SoS or 
FoS (name related programs). 

3. Identify SoS spirals/blocks or other expected increments and their 
timeframes including spirals/blocks of specific systems of the SoS. 
Provide list of expected changes in architecture, performance, 
functionality and technology. 

4. In the SoS TRA include all CTEs required to meet SoS KPPs/operational 
requirements; include SoS unique CTEs as well as system unique CTEs 
required for the specific system to participate as a system of the SoS (e.g., 
a new radio) regardless of who is responsible for funding or developing. 

5. Provide an update to the SoS TRA when any of the systems of the SoS are 
going thru a spiral upgrade independently of the SoS. Each system of the 
SoS needs to be assessed for any changed technology or technology 
implementation to assure SoS perfonnance is preserved. 

6. SoS Milestones B and C shall be scheduled post system Milestone Bs and 
Milestone C’s for SoS (or at least in the same timeframe, +/- 3 months). 
Specific systems need to demonstrate TRL 6 and TRL 7 prior to 
demonstrating SoS TRL 6 and SoS TRL 7. 

7. Systems that are part of a SoS shall include SoS CTEs in their system 
(SoS in the case of a IAMD SoS) specific TRA. Each individual system 
will need to develop their system specific technologies to a TRL 6 and 
above as well as demonstrate system functionality with SoS specific 
technologies to a TRL 6 and above. 
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8. 


All SoS CTEs whether part of the SoS or part of the specific systems of 
the SoS, shall be assessed against SoS requirements. These assessments 
should begin as early as possible (recommend TRL 3). 

In analysis of all four systems, these guidelines were found to be helpful in 

the analysis with respect to technology readiness. If TBMCS had initially followed these 

guidelines they would have had an opportunity to detennine their system type and started 

the appropriate engineering up front. It appears that FCS may still not understand the 

type of system they are building since they indicate they are a FoS enabled by a SoS. 

They have had significant issues with program structure and technology readiness. They 

may have more success if they establish their FCS SOSCOE (or just FCS COE) and 

develop and deploy systems as they are mature. 

AIAMD SOS and SIAP appear to be on track with most of the SoS TRA 
requirements and guidelines. Neither system has pasted MS B; therefore, optimism is 
probably warranted while declaration of success will be based on results of SOS 
development and testing. 

With regard to the extension of the descriptions at a minimum for TRL 3- 
9, this will be the subject of follow-on work. In general, it will be recommended to stay 
consistent with System TRAs and explicitly add language that acknowledges the SoS 
TRA process that a program must show by analysis and then by demonstration and test in 
a SoS environment that the CTE are sufficient to meet functionality, behavior and 
performance wrt the appropriate interoperability attributes in support of SoS KPPs. The 
author used a combination of the current TRL descriptions, MDA checklist and Nolte’s 
calculator (as a checklist) to conduct a now approved Joint SoS TRA for SIAP in support 
of an anticipated MS B. 

b. SoS Technology Location/Identification 

Applying the SoS Technology Location/Identification checklist was not 
able to be assessed by this research. This list was developed and proposed by the author 
while the Technology Development Division Chief at SIAP. This checklist was used to 
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identify technology by the SIAP Technology Development team in collaboration with the 
SIAP SoS systems. Future collaboration with other SoS programs and S&T professional 
is needed. 


3. SoS Acquisition Challenges 

SoS, FoS and Enterprise systems have significant challenges given the 
expectations of increased levels of perfonnance, diversity of systems, and increased 
degrees of interoperability. SoS Engineering is a necessary and time consuming process 
required to achieve success. Acquisition planning and timelines, requirements setting, 
TRA and other acquisition documentation, and proper modeling and simulation/test 
facilities for SoS should be planned up front to obtain the resources required and set 
expectations. The following is a summary of the challenges experienced by the four 
systems analyzed. 


a. SoS Technical Challenges 

The beginning years of a program are extremely difficult if the system 
definition and expected degree of interoperability are not set properly. System 
architecture and engineering activities will not be accomplished (in the case of TBMCS) 
or be conflicted (in the case of FCS). Requirements churn and failure to include critical 
systems inside the lifelines of the FoS/SoS will lead to program restructure and failure. 
FCS has been the subject of GAO reports; this may in fact not be warranted if they 
concentrated on SOSCOE first and then added systems over time. 

System engineering activities with systems outside the lifelines of an 
Enterprise system FoS or SoS will continue to be a challenge given the scope of external 
systems and the unaligned acquisition schedules. FCS has 52 programs outside its 
lifelines that are considered required to meet KPPs and yet they only have 14+1+1 that 
are inside their lifelines. It was probably challenging to justify program personnel to 
adequately cover the system engineering activities required to engineer 66+ systems 
together based on a program of 14+. It’s as if the house is being built and the wiring and 
plumbing will be designed and installed after the fact. 


124 



The JCIDs and CDD process of defining SoS and FoS capability 
requirements provides opportunity to put together systems in innovative and collaborative 
ways to meet the requirements; however, the work required to allocate those KPPs 
appropriately across the SoS is challenging. Its especially challenging when the model 
and simulation is predicated on system models and simulations of medium to high 
fidelity which may not be available for sometime after program initiation. 

b. SoS Systemic Challenges 

Program synchronization appears to be impossible given the number of 
systems that are required to collaborate unless the DoD acquisition model is 
fundamentally changed. It’s extremely difficult to coordinate inside a Service portfolio 
and almost impossible across Services or at the Joint level. Given the nature of systems 
being procured by spiral or blocks, the idea that this level of complexity can actually be 
engineered and managed with the current state of architecture and engineering tools will 
take heroic efforts by talented and experienced professionals. TBMCS and FCS 
indicated they need to at least interface with 76 applications (413 segments) and 170 
systems respectfully - one understands given this number why a COE is so important. 

Given the requirement for MDAP/MAIS technologies to be certified TRL 
6 and the inability to synchronize all the required programs, it may be better to start with 
less systems, initially have lower performance requirements and build the interoperability 
infrastructure first. This would provide for less capability with lower risk. The other 
option is to limit SoS TRA to only the unique CTEs for SoS operations and not expect 
any SoS certifications until the system MSs. This will increase risk to achieving the SoS 
possibly leading to delays in the fielding of the SoS as well as a Nunn-McCurdy breach. 


125 



THIS PAGE INTENTIONALLY LEFT BLANK 


126 



V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

1. System Definitions 

The following definition of SoS from CJCSI 3170 was found to be appropriate for 
conducting a SoS TRA: 

A set or arrangement of interdependent systems that are related or 
connected to provide a given capability. The loss of any part of the 
system will significantly degrade the perfonnance or capabilities of the 
whole. 

The definition was able to be used to distinguish SoSs and non-SoS types of 
systems. Systems were distinguished by whether or not the set of systems were mutually 
dependent and whether their performance would degrade, under a condition where there 
is the minimum number of systems. 

A SoS definition is critical to the development of a SoS for detennining 
acquisition strategies and timelines, requirements, required system engineering activities, 
TRA requirements, acquisition documentation, and proper modeling and simulation/test 
facilities. SoS engineering will be more complex and time consuming than FoS 
engineering given that common distributed processing to enable real-time collaboration 
requires consideration of all systems constraints and restraints within the SoS. The 
research shows that the lack of the proper distinctions as to type of system can lead to 
improper funding and time allocations for development. FCS in particular if handled as a 
FoS vice a SoS would be able to put together a program structure that develops the 
SOSCOE first and then builds out the FCS by adding systems as they mature. FCS 
appears to be attempting to do both a FoS and SoS simultaneously which has put the 
program under undue pressure and assessment. The SOSCOE is an enabler for 
infrastructure functions and would be better labeled a COE. 

Making a distinction of types of SoS may be useful. The current distinctions 
found in the literature review and those proposed in this thesis do not yet assist in 
defining the key features of the SoS. Without the key features being defined, the SoS 
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will be at risk of failure during development and testing in an inappropriate operational 
relevant environment. This occurred for both as TBMCS and FCS. 

2. Interoperability Taxonomy 

The following defined degrees of interoperability were used in the research where 

Command and Control/directing was evaluated in the context of the degree of 

interoperability: 

Level 0 Connectionless 

Level 1 Contribute - situational awareness primarily for safety or information 
exchange 

Level 2 Coordinate - determination of how and when to share resources 

Level 3 Cooperate - determination and plan of how to accomplish related tasks 
for a common mission 

Level 4 Collaborate - detennination and work to be done together to accomplish 
a task 

These levels were found to be appropriate for distinguishing degrees of 
interoperability and were able to be used to verily a SoS or non-SoS. They were useful 
when combined with the interoperability attributes for identifying and locating 
technologies in support a TRA. Key to applying these degrees of interoperability was 
understanding the nature of the required system interactions. If a TBMCS system was 
passing data to another TBMCS system to support mission planning this was clearly 
cooperating towards a mission, whereas, in AIAMD SoS the passing of data to support 
SIAP was clearly a collaboration to determine the air picture via common distributed 
processing (algorithms). Failure to distinguish the level/degree of interoperability led to 
the first TBMCS operational tests failing because of an expectation of collaboration vice 
cooperation. 

The literature review showed no agreed to definitions of degrees of 
interoperability exist. In general, the term collaboration in the literature is used very 
broadly to mean - any coordinated, cooperative, or collaborative effort. The sense is that 
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collaborative is better than coordinated or cooperative. This broad interpretation and 
desire to be ‘collaborative’ has the unfortunate effect of strengthening the practice of 
labeling any set of systems a SoS vice FoS or Enterprise System. As indicated prior, 
misapplication of SoS to FCS has led to program and engineering challenges. FCS is 
specifically cited as saying their minimum requirement is to do as well as the present 
requirement, where systems are not connected via the SOSCOE, leading one to conclude 
that cooperation is the minimum capability required. The assessment of TRLs in a 
cooperative environment may not have CTEs associated with armor be in the same list as 
SOSCOE CTEs such as radios. 

This thesis shows that there are no DoD definitions regarding degrees of 
interoperability even though guidance clearly indicates that interoperability is key to 
current and future warfighting. Taking steps towards defining the levels or degrees of 
interoperability would assist in DoD programmatics and systems engineering activities. 

3. SoS TRA Requirements and Guidelines 

The TRA Deskbook is an excellent guide for conducting TRAs. Adding SoS 
specific requirements and guidelines would facilitate performing SoS TRAs. The 
following SoS TRA requirements and guidelines were used during the research to 
determine if using these guidelines would have benefited the programs analyzed. 

1. Clearly describe the type of SoS and degree of interoperability required - 
(SOA, COE, or CDP) and provide rationale. 

2. Indicate which if any of the systems of the SoS is part of another SoS or 
FoS (name related programs). 

3. Identify SoS spirals/blocks or other expected increments and their 
timeframes including spirals/blocks of specific systems of the SoS. 
Provide list of expected changes in architecture, performance, 
functionality and technology. 

4. In the SoS TRA include all CTEs required to meet SoS KPPs/operational 
requirements; include SoS unique CTEs as well as system unique CTEs 
required for the specific system to participate as a system of the SoS (e.g., 
a new radio) regardless of who is responsible for funding or developing. 
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5. The PM should require an update to the SoS TRA when any of the 
systems of the SoS are going thru a spiral upgrade independently of the 
SoS. Each system of the SoS needs to be assessed for any changed 
technology or technology implementation to assure SoS performance is 
preserved. 

6. SoS Milestones B and C shall be scheduled post system Milestone Bs and 
Milestone C’s for SoS (or at least in the same timeframe, +/- 3 months). 
Specific systems need to demonstrate TRL 6 and TRL 7 prior to 
demonstrating SoS TRL 6 and SoS TRL 7. 

7. Systems that are part of a SoS shall include SoS CTEs in their system 
(SoS in the case of a IAMD SoS) specific TRA. Each individual system 
will need to develop their system specific technologies to a TRL 6 and 
above as well as demonstrate system functionality with SoS specific 
technologies to a TRL 6 and above. 

8. All SoS CTEs whether part of the SoS or part of the specific systems of 
the SoS, shall be assessed against SoS requirements. These assessments 
should begin as early as possible (recommend TRL 3). The technology 
developer should received the SoS requirements that may impact 
technology development as early as possible. 

The SoS TRA requirements and guidelines were found to be appropriate and 
would have been helpful to the programs analyzed. FCS is not synchronized with system 
the other systems that are critical to meet its operational requirements. FCS would have 
been required to have all CTEs at TRL 6 by MS B vice progressing through a CDR 
without all technologies at TRL 6. Without specific guidelines, each DoD program will 
approach SoS and FoS differently and each will go through the learning curve without 
benefit of other programs’ lessons learned. The primary difference between these SoS 
TRA requirements and guidelines vice the ones in the TRA Deskbook is the specifics for 
what technologies to include in the SoS TRA and the timing of SoS MSs and TRAs. 
These requirements and guidelines are extensions of a system TRA requirements and 
guidelines. The biggest impact to a program is that technologies will most likely need to 
be matured earlier in their program in order to be demonstrated in a SoS environment. 


130 



4. SoS Technology Location/Identification Checklist 

A SoS technology identification and locator checklist was proposed from the 
literature review and used during the research analysis. The SoS checklist is focused on 
identification and location of technologies based on interoperability attributes and 
architecture and system engineering artifacts. This checklist and Nolte’s TRL calculator 
was used by the author during SoS TRA activities for SIAP and was found to be useful in 
identifying and locating technologies within each of the systems of the SoS. An example, 
it was used during the research to evaluate available FCS’s engineering artifacts, this led 
to the conclusion that the CTE list may be incorrect given the systems may only be 
required to be cooperative (vice collaborative). 

The literature is rich with SoS architecture assessment approaches, these coupled 
with Nolte’s TRL calculator can be used for developing a SoS technology 
identification/locator checklist. The author is interested in future collaboration with other 
SoS programs and S&T professionals to determine and develop a technology 
identification and location checklist for SoS. 

5. SoS Acquisition Challenges 

Technical challenges include a) performing SoS engineering activities prior to 
system engineering activities and many SoS are assembled from legacy systems, b) KPPs 
for a capability are not easily allocated to individual systems, c) appropriate SoS relevant 
environment modeling and simulation and test and evaluation environments will typically 
be built post system design and development, d) identification of SoS CTEs, and e) SoS 
are typically enabled with software which is easily changed. Given these challenges, the 
technology development and acquisition strategies for the researched potential SoS were 
assessed for their ability to be employed for SoS technology maturation given the 
challenges of synchronization. 

The research showed that SoS architecture, technology development and 
engineering activities need to be in alignment in order to reduce the risk to the SoS 
development. Identification, location and development of common elements and 
common distributed processing require SoS engineering upfront by all parties involved. 
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TBMCS found that they were able to make progress once requirements were set and 
system engineering activities were formalized. FCS is in the process of determining final 
system requirements and their program has be restructured to reflect the systems that are 
maturing in the first spiral. IAMD SOS and SIAP are being synchronized and SoS 
modeling and simulation and test events are being put in place to properly demonstrate 
and test to SoS requirements. Without these adjustments to these programs, the programs 
would experience failure. 

There is not enough research or data to analyze regarding how the perfonnance of 
systems of a SoS together will meet the SoS KPPs/operational requirements. More 
research will need to be accomplished as these systems are developed and tested. 

There is not enough research or data to analyze the effect of spiral or block 
development on SoSs. Configuration control and management will be key to the success 
in this area. 

SoS architecture, engineering and a SoS TDS will be key to developing SoS 
technologies. Innovation in S&T and acquisition strategies will be required to 
successfully develop SoS. 

Systemic challenges within the DoD include: a) critical technology developed by 
the individual programs are in alignment with their respective schedules not the SoS 
program schedule, b) SoS technology selections and development prior to completion of 
capability engineering and then individual system(s) engineering drives up risk; SoS 
engineering needs to be at least through System Functional Review prior to a MS B 
decision, and c) it's challenging to test the critical technologies in an integrated manner if 
the individual systems have not had the opportunity to all develop their systems enough 
to have representative systems for SoS testing (e.g., relevant environment for a integrated 
heterogeneous distributed system) and d) the fielding of a SoS capability is typically 
time-phased over several years in capability spirals or increments with differing sets of 
systems and services. 

The AIAMD SoS and SIAP program are moving towards aligned system 
schedules for key systems. The other systems of these SoS have been phased to occur at 
later dates. Direction and funding was applied to SIAP systems up front to perfonn the 
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required SoS engineering activities including accomplishing a SoS System Functional 
Review and SoS Preliminary Design Review. AIAMD SOS and SIAP included key 
systems only for the first spiral of their respective SoS. The SoS engineering accelerated 
system architecture and engineering activities for the systems of the SoS; however, the 
development of technologies and systems was not able to be accelerated due to the 
funding levels being normally phased later for the systems. This impacts the ability to 
perform SoS technology demonstration without up-front planning. 

System development occurs in alignment with system program plans and funding; 
this puts the schedules for SoS Developmental Testing (DT) and Operational Test and 
Evaluation (OT&E) post the systems DT and OT&E. Some systems will achieve a 
system Initial Operating Capability (IOC) prior to the SoS IOC. SoS DT events will need 
to take place prior to approving a limited fielding of the system’s SoS capability. After 
an initial synchronization of system schedules, follow-on spirals or blocks will most 
likely be out of synchronization with follow-on spirals or blocks of the other systems of 
the SoS. It is anticipated that the AIAMD and SIAP schedules which are synchronizing 
during the first spiral will be unsynchronized for subsequent spirals. For COEs, the 
research shows that they can be developed and then systems which conform to the COE 
standards and protocols can be phased with little impact. 

System engineering activities with systems outside the programmatics of a SoS 
will continue to be a challenge given the scope of external systems and the unaligned 
acquisition schedules. TBMCS, FCS and AIAMD SOS have a number of significant 
technologies being developed outside of their program cycles and this synchronization 
issue doesn’t appear to have been successfully addressed. Research is required to 
determine when and if a SoS is really required. Also, research is required to determine if 
architectures, standards and protocols if implemented properly will mitigate the need to 
have SoS which are tightly coupled in order to accomplish task synchronization. 

Based on the research it is recommended that a SoS be formally initiated with 
requirements and designated technology development and acquisition strategies. Funding 
and direction should be provided to ah programs required to participate in the SoS to 


133 



accomplish SoS architecture and engineering activities prior to system development. 
Implementation by ah the systems may be able to be time-phased. 

SoS Milestones B and C shah be scheduled post system Milestone Bs and 
Milestone C’s (or at least in the same timeframe, +/- 3 months). Systems need to 
demonstrate TRL 6 and TRL 7 prior to demonstrating SoS TRL 6 and SoS TRL 7. 

B. RECOMMENDATIONS AND FUTURE RESEARCH 

1. System Definitions 

It is recommended that all of DoD adopt the same SoS and FoS definitions and 
that USD(AT&L) use these definitions in their DAG, TRA Deskbook, SoS Engineering 
Guide and other S&T and acquisition directives and instructions. This would provide for 
commonality of efforts and expectations for SoS technology development and 
acquisition. 

Further research and definition into the types of SoS should be pursued given 
normally one size doesn’t fit ah, this would enable proper SoS program planning and 
execution. 

2. Interoperability Taxonomy 

It is recommended that DoD, including the Chief of the Joint Chiefs of Staff and 
USD(AT&L) develop and adopt a common taxonomy for degrees of interoperability in 
acquisition guidance in support of the JCIDS process. It is recommended that the degree 
of interoperability taxonomy be based on nature of the communications to support 
mission and common tasks and subsequently required perfonnance be defined in terms of 
the Interoperability Attributes (e.g., accuracy, latency). 

Also, it is recommended that the Milestone Decision Authority detennine and 
designate require key OVs and SVs be accomplished prior to MS decisions. 

3. TRAs Requirements and Guidelines 

It is recommended that DUSD(S&T) include specific SoS requirements and 
guidelines in the TRA Deskbook. Adding SoS specific requirements and guidelines 
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would facilitate performing and evaluating SoS TRAs. Also, it is recommended that as 
SoS types are identified that these are included in the TRA Deskbook vice the current 
listing of IT systems. 

Also, the TRL descriptions do not currently describe SoS aspects. It is 
recommended that specific language is added regarding recommended SoS 
demonstrations and testing in the descriptions for both hardware and software TRL. 

4. SoS Technology Identification/Location Checklist 

A SoS technology identification and locator checklist was proposed from the 
literature review and used during the research analysis. It is recommended that a SoS 
checklist be developed in collaboration with other SoS programs and S&T professionals 
to determine and develop a technology identification and location checklist for SoS. 
Nolte’s calculator is a great start towards such a tool. This checklist would be a useful to 
include in an appendix of the TRA Deskbook. 

5. SoS Technology Development and Acquisition Strategies 

It is recommended that the Milestone Decision Authority hold a formal Program 
Initiation meeting to start technology development for SoS unique technologies and to 
begin SoS architecture and engineering activities with the requisite direction and funding. 
It is recommended that all anticipated systems be directed to participate in the SoS 
architecture and engineering activities and then time-phase these systems (no big bang 
with tens of systems). In addition, it is recommended that if a COE is needed that it be 
engineered first, prior to development of systems that use the COE. Also, it is 
recommended that SoS Milestones B and C are scheduled post systems Milestone Bs and 
Milestone C’s (ok if in the same timeframe, +/- 3 months). 

It may be that defined architectures, standards and protocols if implemented 
properly would mitigate the need to have tightly coupled systems such as a SoS that 
requires common distributed processing. It is recommended that research be conducted 
to detennine what SoS are truly needed and when COE, FoS or Enterprise System would 
meet warfighting requirements, since SoS acquisition is challenging and expensive. 
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